Miranda API doesn't seem to work, but WinInet does?

[omniwolf]

Hi Guys,
i hope this makes sense. doing a GET with miranda API is fine, but when i do a POST using Miranda API:

	meth = http::post;
	std::string auth_STR = "my auth code";
	std::string pdata_STR = "status=test";

	NETLIBHTTPHEADER hdr[2];
	hdr[0].szName = "Authorization";
	hdr[0].szValue = const_cast<char*>(auth_STR.c_str());

	req.headers = hdr;
	req.headersCount = 1;

	if(meth == http::post)
	{
		hdr[1].szName = "Content-Type";
		hdr[1].szValue = "application/x-www-form-urlencoded";

		req.headersCount = 2;
		req.dataLength = pdata_STR.size();
		req.pData = const_cast<char*>(pdata_STR.c_str());
	}

	http::response resp_data;

	resp = reinterpret_cast<NETLIBHTTPREQUEST*>(CallService( MS_NETLIB_HTTPTRANSACTION,
		reinterpret_cast<WPARAM>(handle_), reinterpret_cast<LPARAM>(&req) ));

it doesn't work. i get "invalid signature" return data..
however, if i use WinInet, it works. like this:

	wstring auth = "my auth code";
	oauthHeader = L"Authorization: " + auth + L"\r\n";
	std::string post_data = "status=test";
	wstring httpMethod = L"POST";

	wchar_t host[1024*4] = {};
	wchar_t path[1024*4] = {};

	URL_COMPONENTS components = { sizeof(URL_COMPONENTS) };

	components.lpszHostName = host;
	components.dwHostNameLength = SIZEOF(host);

	components.lpszUrlPath = path;
	components.dwUrlPathLength = SIZEOF(path);

	wstring normalUrl = url;

	BOOL crackUrlOk = InternetCrackUrl(url.c_str(), url.size(), 0, &components);
	_ASSERTE(crackUrlOk);

	wstring result;

	// TODO you'd probably want to InternetOpen only once at app initialization
	HINTERNET hINet = InternetOpen(L"tc2/1.0", 
		INTERNET_OPEN_TYPE_PRECONFIG, 
		NULL, 
		NULL, 
		0 );
	_ASSERTE( hINet != NULL );
	if ( hINet != NULL )
	{
		// TODO add support for HTTPS requests
		HINTERNET hConnection = InternetConnect( 
			hINet, 
			host, 
			components.nPort, 
			NULL, 
			NULL, 
			INTERNET_SERVICE_HTTP, 
			0, 0 );
		_ASSERTE(hConnection != NULL);
		if ( hConnection != NULL)
		{
			HINTERNET hData = HttpOpenRequest( hConnection, 
				httpMethod.c_str(), 
				path, 
				NULL, 
				NULL, 
				NULL, 
				INTERNET_FLAG_KEEP_CONNECTION | INTERNET_FLAG_NO_CACHE_WRITE | INTERNET_FLAG_NO_AUTH | INTERNET_FLAG_RELOAD, 
				0 );
			_ASSERTE(hData != NULL);
			if ( hData != NULL )
			{
				BOOL oauthHeaderOk = HttpAddRequestHeaders(hData, 
					oauthHeader.c_str(), 
					oauthHeader.size(), 
					0);
				_ASSERTE(oauthHeaderOk);

				// NOTE POST requests are supported, but the MIME type is hardcoded to application/x-www-form-urlencoded (aka. form data)
				if(Compare(httpMethod, L"POST", false) && post_data.length() > 0)
				{
					wstring contentHeader = L"Content-Type: application/x-www-form-urlencoded\r\n";
					BOOL contentHeaderOk = HttpAddRequestHeaders(hData, 
						contentHeader.c_str(), 
						contentHeader.size(), 
						0);
					_ASSERTE(contentHeaderOk);
				}

				BOOL sendOk = HttpSendRequest( hData, NULL, 0, (LPVOID)(post_data.size() > 0 ? post_data.c_str() : NULL), post_data.size());
				_ASSERTE(sendOk);

				// TODO dynamically allocate return buffer
				BYTE buffer[1024*32] = {};
				DWORD dwRead = 0;
				while( 
					InternetReadFile( hData, buffer, SIZEOF(buffer) - 1, &dwRead ) &&
					dwRead > 0
					)
				{
					buffer[dwRead] = 0;
					result += UTF8ToWide((char*)buffer);
				}
				InternetCloseHandle(hData);
			}
			InternetCloseHandle(hConnection);
		}
		InternetCloseHandle(hINet);
	}

	return result;

it works! is there something different between the way it's done between the two code snippets? please let me know if you need more info
thanks in advance

[borkra]

Hi Guys,
i hope this makes sense. doing a GET with miranda API is fine, but when i do a POST using Miranda API:

it works! is there something different between the way it's done between the two code snippets? please let me know if you need more info
thanks in advance

Yes where is your network log?

In Wininet code I see

(post_data.size() > 0 ? postDataUTF8.c_str() : NULL), post_data.size());

Nothing like that in Miranda code... Why?

Content type is "application/x-www-form-urlencoded"
I do not see you url encode just anything. Obviously server will error out.

It's not obvious if you are using the same url in Miranda or not.

If you want somebody to debug your project for you you need to send the whole project.

[omniwolf]

Yes where is your network log?

it's not that interesting, the wininet code obviously doesn't output anything to the network log, but the miranda code gives this:

[1:29:24 Twitter_1] **SLURP::POST - req.pdata is status=testtest
[1:29:24 Twitter_1] Connecting to twitter.com:80 (Flags 1)....
[1:29:24 Twitter_1] (1144) Connected to twitter.com:80

[1:29:24 Twitter_1] (05A4AFF0:1144) Data sent
POST /statuses/update.json HTTP/1.0
Authorization: OAuth oauth_consumer_key="AygxrTNGti27uVMz0hAbA",oauth_n once="P1kAlMiG2Kb7F
zP5t",oauth_signature="DmeyKRQukLAUkh%2Fy71TjDyPtF M4%3D",oauth_signature_method="HMAC-SHA1
",oauth_timestamp="1284218963",oauth_token="770845 91-qIzIZhsFTBJjTLBJ9CbCUOgHQbjDfS7szngxU
GG0",oauth_version="1.0"

Content-Type: application/x-www-form-urlencoded
User-Agent: Miranda/0.9.2 (Unicode)
Accept-Encoding: deflate, gzip
Host: twitter.com
Connection: Keep-Alive
Proxy-Connection: Keep-Alive
Content-Length: 15


[1:29:24 Twitter_1] (05A4AFF0:1144) Data sent
status=testtest

[1:29:24 Twitter_1] (05A4AFF0:1144) Data received
HTTP/1.1 401 Unauthorized

[1:29:24 Twitter_1] (05A4AFF0:1144) Data received
Date: Sat, 11 Sep 2010 15:29:24 GMT
Server: hi
Status: 401 Unauthorized
WWW-Authenticate: Basic realm="Twitter API"
Content-Type: application/json; charset=utf-8
Content-Length: 65
Cache-Control: no-cache, max-age=300
Set-Cookie: k=124.171.21.131.1284218964480735; path=/; expires=Sat, 18-Sep-10 15:29:24 GMT
; domain=.twitter.com
Set-Cookie: guest_id=128421896448528189; path=/; expires=Mon, 11 Oct 2010 15:29:24 GMT
Set-Cookie: _twitter_sess=BAh7CDoPY3JlYXRlZF9hdGwrCBJKagErAToH aWQiJTYxNjk0OWJkOWNjNThl%250
AOWNlMjU3NzJhNjIwMTEwOTc2IgpmbGFzaElDOidBY3Rpb25Db 250cm9sbGVy%250AOjpGbGFzaDo6Rmxhc2hIYXNo
ewAGOgpAdXNlZHsA--ce6721efa68d17a73a4ed484cd51c42906262be6; domain=.twitter.com; path=/
Expires: Sat, 11 Sep 2010 15:34:24 GMT
Vary: Accept-Encoding
X-XSS-Protection: 1; mode=block
X-Frame-Options: SAMEORIGIN
Connection: close


[1:29:24 Twitter_1] (05A4AFF0:1144) Data received
{"request":"/statuses/update.json","error":"Incorrect signature"}

[1:29:24 Twitter_1] (05A4AFF0:4294967295) Connection closed

In Wininet code I see

(post_data.size() > 0 ? post_data.c_str() : NULL), post_data.size());

Nothing like that in Miranda code... Why?

well, as i said at the start, i'm only concerned with the POST stuff, so the equivalent in the miranda API code is the "if meth == http::post" block. the code you posted above, the post_data will always have stuff in it.

Content type is "application/x-www-form-urlencoded"
I do not see you url encode just anything. Obviously server will error out.

It's not obvious if you are using the same url in Miranda or not.

i am url encoding everything, i specifically used a string that didn't need to be url encoded in my example so we didn't need to worry about it. it's the same URL.. i run both bits of code sequentially in my project with exactly the same input, and the miranda api one fails :(

If you want somebody to debug your project for you you need to send the whole project.

sorry that i'm annoying you :(

[borkra]

One of the problems I see you use HTTP 1.0. It's probably not going to work. You need to use HTTP 1.1 with persistent connections and redirects. Because authentication consist of multiple http transactions that have to be done over the same connection. Take a look on MSN or Weather how persistent http is done.

You need to add 3 flags:
NLHRF_HTTP11
NLHRF_PERSISTENT
NLHRF_REDIRECT

And manage connection handle -

pass connection handle returned by one http request to the other.
And close connection when you done.

[omniwolf]

hi borkra, thanks for your reply.
how did you work out those flags that i need? is it from the "INTERNET_FLAG_KEEP_CONNECTION | INTERNET_FLAG_NO_CACHE_WRITE | INTERNET_FLAG_NO_AUTH | INTERNET_FLAG_RELOAD" code in the WinInet part of my code?

i added those 3 flags that you mentioned to my code, but it seems as soon as i add NLHRF_HTTP11 as a flag, i get this in my network log:

[15:39:10 Twitter_1] (006A29B0:776) Data received
HTTP/1.1 400 Bad Request

[15:39:10 Twitter_1] (006A29B0:776) Data received
Date: Tue, 14 Sep 2010 05:39:10 GMT
Server: Apache
Vary: Accept-Encoding
Content-Length: 226
Connection: close
Content-Type: text/html; charset=iso-8859-1


[15:39:10 Twitter_1] (006A29B0:776) Data received
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>400 Bad Request</title>
</head><body>
<h1>Bad Request</h1>
<p>Your browser sent a request that this server could not understand.<br />
</p>
</body></html>

i had a look in your msn and weather plugin code, as far as i can tell it does pretty much the same stuff as mine.. after you do the CallService(HTTPTRANSACTION) call, assign a bunch of the reply struct members to new variables, and then free the reply struct.
but in your post above you say that i have to "manage the connection handle" by "passing the returned handle from one http request to the other".. what do you mean "the other"? i'm only making one HTTP POST request here aren't i? or am i misunderstanding something?

ok i just had another look at your ssl code in MSN, i see at the start of the function (before CallService(HTTPTRANSACTION)), you do this:

nlhr.nlc = hHttpsConnection;

and then after you get the response from CallService(HTTPTRANSACTION), you do a

hHttpsConnection = nlhrReply->nlc;

so i added in a new handle to my code

HANDLE httpPOST_;

but as soon as i put:

req.nlc = httpPOST_;

in my code, CallService(HTTPTRANSACTION) doesn't give me a response... any ideas? i don't really understand the idea behind these HANDLE things.. i seem to be passing them around the code arbitrarily.. is it bad that i used a different name to yours? (hHttpsConnection)

[borkra]

Your Network log shows response but not the request and it is needed.

Handle is plugin link to allocated Miranda core resources.

In HTTP 1.0 clients connects, performs http transaction and disconnects, this makes it slow and makes you unable to perform authentication as connection needs to stay between calls. HTTP 1.1 allows to have persistent connections, so you connected to server once and then just perform http transactions on existing connection. Thus when you perform Miranda http transaction you need to tell it that you are already connected to the server (pass valid connection handle from one transaction to the other). On the very first transaction or when you do not have valid connection handle it needs to be set to NULL. When you longer need open connection you need to close handle.

Again it's nearly impossible to understand what's going on with your authentication without playing around with it. Send me code for the plugin, I will look at it.

[omniwolf]

Just thought i would post how this was fixed in the end:

in my Auth string i appended a "/r/n" to the end of it.. this was required when using WinInet. however with miranda the auth header should just be the string. why this didn't break GET requests but did break POST requests i don't understand, but now both work.

thanks to borkra for having a look at my code.

[borkra]

why this didn't break GET requests but did break POST requests i don't understand, but now both work.

Oh no it broke both but in a different manner, as POST allowed to have data and GET not.

[joem]

Hi everyone,
does anyone know whether HTTP/1.0 or HTTP/1.1 is fine with Polipo? As far as i know it Polipo does not support the long-obsolete HTTP/0.9.

[borkra]

1.1 - is latest version of protocol ... In general you should not be using IM with HTTP proxies. IM is not a web browser, proxies do nothing for IM but degrade the protocol.