XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

[paqx]

Hi!

As you might know, there are currently white lists for mobile internet in many regions of Russia. It's hard set up an xray server to bypass the restriction because you need either a white-listed IP or a white-listed CDN that supports POST requests. This commit targets the latter case. There are few white listed CDNs in Russia and unfortunately they try to block xray requests. I myself bumped into one such case. CDNVideo started to block any POST requests whose query string matches the pattern x_padding=XXXXX. Also, some requests included an X-Padding header. Both the query param and header have a very unusual name and can be easily spotted. My goal was to bypass this by allowing users to change the name of the query param. I ran a test and successfully bypassed CDNVideo's filter. It's not a perfect solution because they can start blocking requests by the value but anyway.

What I did:

1. Added xPaddingQueryParam. It can be set in the config to change the name of the query param used to pass the padding data. The default is x_padding
2. Renamed X-Padding header to X-Signature - a lot more common and widely used by actual CDN users.
3. Changed the way the padding data is generated. It now consists of chars X and Z, not only X.

Further improvements:

• Other CDNs disabled POST requests but PUT and PATCH requests are often not blocked. I plan to allow users to choose an HTTP method to pass data.
• The padding value is still very noticable. I don't understand the idea well enough yet but it the post said X was chosen because it's compressed into 8 bits. I checked the encoding table and the only suitable alternative was char Z. Not much and I wonder if it's possible to make the data look more random. In this case it will be very hard to block it because it can disguise as cache busters.
• Use a more realistic user agent. I noticed that the requests are currently passed with the Go-http-client/2.0 UA. Also doesn't look like a real CDN user.

Thanks

[Fangliding]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)

不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

[hxehex]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)

不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

I thought the pr was in english, and you are replying in chinese. where is your conscience or respect? what were you thinking when making this reply and then closing this pr?

and then, you did not give an exact valid reason on why you closed this pr. all you said is that you can handle yourself the random generation of something I still don't understand. you ignored the main purpose of this pr, that is clearly in it's title:

XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

..or if you can't read english, here you go:

XHTTP：将X-Padding重命名为X-Signature，新增xPaddingQueryParam参数

the #4346 issue you pointed to is dead. nothing was made or changed since (1 month) the last reply in it (not counting mine reply)

[hxehex]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

[RPRX]

我们看得懂英文，回复用中文纯粹是因为对我们来说码字速度更快，没有别的意思

我觉得从你改了多个部分的代码以及“Further improvements”的思考来看，你对 Xray 的架构有不浅的研究、完全理解 XHTTP 的工作原理、且有发散性的思维，也就是说你是一个持续改进 XHTTP 的不错的人选，所以我会 reopen 这个 PR，只是怎么改还需讨论

就像 Vision Seed 一样，本来在等 GFW 动手，如果 XHTTP 再不加自定义的话等下隔壁出个类似的传输层号称啥都能改，绷不住

Renamed X-Padding header to X-Signature - a lot more common and widely used by actual CDN users.

修改默认名称会破坏 XHTTP 新旧版本之间的兼容性，应当加选项

Added xPaddingQueryParam. It can be set in the config to change the name of the query param used to pass the padding data. The default is x_padding
Changed the way the padding data is generated. It now consists of chars X and Z, not only X.

然后关于这两点我思考过 #4346 (comment) ，如果要改应当加上七八个选项，或者就干脆不带 padding，你觉得哪一种更好？

[RPRX]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

我突然觉得 Xray 这名字挺贴切的，对不同墙的穿透性不同，但毕竟不是 Gamma-ray，尽力而为，如果完全断网那只能星链

[hxehex]

also, the xray phrase "Xray, Penetrates Everything." right now is invalid - because it does not "penetrate" CDNvideo

我突然觉得 Xray 这名字挺贴切的，对不同墙的穿透性不同，但毕竟不是 Gamma-ray，尽力而为，如果完全断网那只能星链

🫤 starlink is not available nor it does work in Russia (officially)

[hxehex]

修改默认名称会破坏 XHTTP 新旧版本之间的兼容性，应当加选项

I guess neither me or @paqx thought about that

I think maybe we should revert that one change but still allow customizing the header name

[Fangliding]

这个部分暂时不需要pr 不过它是计划中的 #4346 (comment)
不用X换用随机ASCII字符的方法非常简单 只需要确定字符集(eg base62)然后计算它在哈夫曼编码下的压缩率然后将预期的xpadding长度除以它做修正就行了 最后的结果会趋近xpadding的长度而且还会额外附加一点随机性 这早在全部用X填充之前我就想到了 不过问题不大而且这个字母占8个位确实很有缘所以当初就没提这一点(当然复杂一点的话精确长度也是完全可以做到的)

I thought the pr was in english, and you are replying in chinese. where is your conscience or respect? what were you thinking when making this reply and then closing this pr?

and then, you did not give an exact valid reason on why you closed this pr. all you said is that you can handle yourself the random generation of something I still don't understand. you ignored the main purpose of this pr, that is clearly in it's title:

XHTTP: rename X-Padding to X-Signature, add xPaddingQueryParam

..or if you can't read english, here you go:

XHTTP：将X-Padding重命名为X-Signature，新增xPaddingQueryParam参数

the #4346 issue you pointed to is dead. nothing was made or changed since (1 month) the last reply in it (not counting mine reply)

我完全看得懂英文 反正很多人都要借助翻译软件译成自己的母语我还不如也直接敲中文尽可能保留原始意思
我打一开始就看了这些更改没什么很大的建设性 重命名header的名字和把里面填充的字母从全X改成XZ混合并没有什么意义 要检测也就改几行规则的事情 改名字还顺便破坏兼容性 但是这个header确实打算开放修改 我才贴了那条link

[hxehex]

@paqx will reply about this problem a bit later he can't answer right now

[paqx]

@RPRX

Thank you for reopening the PR. Russian users really need these changes at the moment to bypass censorship.

Based on your modifications to multiple parts of the code and your consideration of "Further improvements," I believe you have a deep understanding of Xray's architecture, a complete grasp of XHTTP's working principles, and divergent thinking. In other words, you are a good candidate for continuous improvement of XHTTP. Therefore, I will reopen this PR; however, how to modify it still needs discussion.

Thanks, but actually I do web developing in PHP/Python + DevOps. Yesterday was the first day I started to dig the xray code and I haven't coded in the go programming language before. Therefore, I hope you will help me implement the things I outlined in the first post.

Changing the default name will break compatibility between new and old versions of XHTTP; an option should be added.

I will return X-Padding as the default header name for backward compatibility and try to make it possible to change the header name via the config file.

Then I thought about these two points #4346 (comment) . If we were to change it, we should add seven or eight options, or just remove the padding altogether. Which do you think is better?

Could you please explain the idea behind the seven or eight options? In my PR, I implemented the possibility to set any query param name instead of x_padding. Doesn't this allow users to set any name and not chose from just 7-8 options? Or do you mean something else?

As I mentioned, the xray code is new to me. I don't yet understand why some decisions were made. I gathered my knowledge about XHTTP mainly from this post. My current understanding is that the x_padding query param as well as the X-Padding header were added to make the size of HTTP requests more random (so that censors cannot easily detect HTTP requests of the same length and block them). If that's true, I don't think it's a good idea to completely remove the padding. I realize that the primary goal of xray is to bypass censors and not CDNs. However, CDNs in Russia decided to cooperate with censors so I think making xray less detectable for CDNs is necessary too (but not by making it less stealthy for censors).

@Fangliding

Renaming the header and changing the padding from all Xs to a mix of XZs is meaningless. Detection would only require changing a few lines of rules. Renaming it also breaks compatibility. But this header is indeed intended to be open for modification, which is why I posted that link.

I agree that my current changes do dot provide enough protection but I needed to start from something. It currently works and allowed me to bypass the CDNVideo filters. They simply added some regex that searches for x_padding query param followed by a sequence of Xs.

I only changed Xs to a sequence of X+Z because this post specifically mentions that it was chosen since it gets compressed into 8 bits and it sounded like it's important for xray operation. And the only suitable letter that gets compressed into the same 8 bits is Z. So I used what I could use. It would great if you could explain why the padding data is generated like a sequence of Xs now.

Obviously if CDNVideo notice some other param followed by a sequence of Z+X, they will easily block it again. But if I manage to change the way the padding data is generated and make it look random like some hash, it will be a lot harder for them.

For example, real CDN users can often use params like _dc for cache busting and if the padding looks something like this:

_dc=iubwefwe2423bfwebnklewb2b23t23

I doubt they will decide to block it because it might cause serious issues for many real users. Isn't it what XHTTP is all about?

The method of using random ASCII characters instead of 'X' is very simple. You just need to determine the character set (e.g., base62), calculate its compression ratio under Huffman coding, and then divide the expected xpadding length by this ratio to make the adjustment. The final result will approximate the xpadding length and also add a bit of randomness. I thought of this before using all 'X's for padding, but it wasn't a big problem, and the fact that this character occupies 8 bits was quite coincidental, so I didn't mention it at the time (of course, a more complex method could achieve the exact length).

I will try to implement this in the PR.

Thanks.

[gfw-killer]

I think the new options could be

1. the padding position (query-string / header / cookie / multipart)
2. it's key-name
3. padding character-set (repeated X or random A-Z,a-z,0-9)
4. upload method (POST / PUT / PATCH)
   it is also possible to upload data using GET request but not all webservers supports it, same about OPTIONS/TRACE/CONNECT/HEAD
   There is also 'Resumable Upload APIs' like Tus; https://developers.cloudflare.com/stream/uploading-videos/resumable-uploads/ (with no need to gRPC support))

[gfw-killer]

I remember there was another obvious characteristic too
POST /yourpath/sameUUID/seq
the connection uuid could also have different position and forms, and also the seq