Is there anything in SSI.php that could interfere with a cURL operation?

Started by rcane, July 16, 2022, 11:43:37 AM

Previous topic - Next topic

rcane

I'm doing a simple curl for members that pulls their zipcode & zip4 from the themes table and dump it here:   https://ziplook.house.gov/zip/ziplook.html

*background*:  The zip4 is used to narrow down your congressional district, in the case where two or more districts might lay across a particular zip code.  Happens quite a bit. 


1. I have tested my code out on an external editor and it performs fine. 
2. But, when I use it on a page that's inside my SMF (where I validate $user_info['groups']) it ignores the zip4 as if it were never entered.  I'm testing on a known zipcode that has 3 districts overlapping.

This happens whether I call the db or just manually assign the variables to test.

Someone suggested it might be something in the SSI.php since it's being required early on.   I'm not sure what that could be, but figured I'd ask if there is anything in there that might be interfering with curl.

Arantor

No, there's nothing in SSI.php that affects cURL.

When you say "as if it were never entered", where is it entered? Is it stored somewhere?

Also, you *know* by now that you get better support by actually posting the code you're using rather than making us guess what you might have written.

rcane

Quote from: Arantor on July 16, 2022, 11:53:15 AMNo, there's nothing in SSI.php that affects cURL.

When you say "as if it were never entered", where is it entered? Is it stored somewhere?

Also, you *know* by now that you get better support by actually posting the code you're using rather than making us guess what you might have written.

I know I know. I was hoping the subject alone might have stuck out. 

it's possible the code just wasn't displaying properly on the online tester and I just "thought" I had it right.

$userzip and $userzip4 are commented out, but I was removing the call to the table and just assigning them the same values (to rule out mySQL) and the results were the same.


error_reporting(E_ALL);
ini_set("error_reporting", E_ALL);


//LET'S GET YOUR ZIP AND 4 FROM THE DB

//get user ID and your State
    $who = $user_info['id'];
    $gather = $conn->query("
                        SELECT
                        max(case when (smfqg_themes.variable like 'cust_zipcod') then smfqg_themes.value else NULL end) as 'cust_zipcod',
                        max(case when (smfqg_themes.variable like 'cust_zip4') then smfqg_themes.value else NULL end) as 'cust_zip4'
                        FROM smfqg_themes
                        where id_member=('$who')");
                       
    if ($gather) {
       
         while($rows = $gather->fetch_object()) {
            $results[] = $rows;
            }
           
         $codes = json_decode( json_encode($results), true);
       
        echo '<br>';
       
            foreach ($codes as $zips) {
                $userzip = $zips['cust_zipcod'];  //assigning the variables for CURL
                $userzip4 = $zips['cust_zip4'];   //assigning the variables for CURL
            }
        }   else {
        echo ("your query is broken");
    }

//###########   zip gets are done ############

$url='https://ziplook.house.gov/htbin/ziplook_find';

// https://ziplook.house.gov/htbin/ziplook_find?773573016773573016


//$userzip = "77357"; in the case of manual testing
//$userzip4 = "3016"; in the case of manual testing

$user_thru = $userzip4; //4-digit range end (not needed)


$curl = curl_init();
$fields[] = 'Content-type: application/json';
$fields = array(
    'zip' => $userzip,
    'zip4' => $userzip4,
    'thru_zip4' => $user_thru
    );


$fields_string = http_build_query($fields);
curl_setopt($curl, CURLOPT_URL, $url);
curl_setopt($curl, CURLOPT_POST, TRUE);
curl_setopt($curl, CURLOPT_POSTFIELDS, $fields_string);
curl_setopt($curl,CURLOPT_RETURNTRANSFER, TRUE);
$data = curl_exec($curl);
curl_close($curl);

$cleanhtml=preg_replace('/[\x00-\x1F\x7F-\xFF]/', '', $data);

//prepare for display

$dom = new DOMDocument();
$dom->loadHTML($cleanhtml);
$dom->preserveWhiteSpace = false;

 $tables = $dom->getElementsByTagName('table');
   $rows = $tables->item(0)->getElementsByTagName('tr');
   $tds = $tables->item(0)->getElementsByTagName('td');
   $link='';
   foreach ($rows as $row) {
      $cols = $row->getElementsByTagName('td');
      $links = $row->getElementsByTagName('a');
        foreach ($links as $link){
          //echo $link->getAttribute('href')."<br>";
         $link= $link->getAttribute('href');
        }


      if($cols->item(0)->nodeValue !=''){
         
      echo 'ZIP CODE: '.$cols->item(0)->nodeValue.'<br />';
      echo 'MEMBER: '.$cols->item(1)->nodeValue.'<br />';
      echo 'MEMBER-LINK: '.$link.'<br />';
      echo 'STATE/DISTRICT: '.$cols->item(2)->nodeValue.'<br />';
      echo 'ROOM: '.$cols->item(3)->nodeValue.'<br />';
      echo 'PHONE: '.$cols->item(4)->nodeValue.'<br />';
      echo '<hr />';
      }
   
   }

rcane

If I remove the two lines that require ssi.php and my db_conn.php, and just run the page without validating a user (and manually assigning the zip variables) it works fine. 

If I re-introduce either line--even without "using" the db connection--it ignores the zip4 and shows ALL the congressman.

The error log also points to a problem with this line--even during the above "..it works fine."


if($cols->item(0)->nodeValue !=''){
There is a bit in SSI.php about making it php compatible.  Is it possible something in there is/isn't getting considered properly?



You cannot view this attachment.

Arantor

OK, I'm deliberately going to not get into the fact that your SQL is doing some pretty hokey things it probably shouldn't ever be doing (why the MAX() against the themes table? That's... unique I guess), I suspect the bigger problem is that you're just making the cURL URL wrong.

I don't know if you know what this code is *actually* doing. It's probably not what you think it is, based on the comments you have in your code.

Your comment suggested that https://ziplook.house.gov/htbin/ziplook_find?773573016773573016 is a valid URL - and this produces a meaningful result if I go there.

If I construct the URL that you construct with that cURL, creating a POST that submits a form with separate parameters I just get an error.

It seems to me that what you need to do is create a URL of:
$url . $userzip . $userzip4 . $userzip . $user_thru
using your variables and change the cURL call to a GET rather than a POST which means dropping the CURLOPT_POST and CURLOPT_POSTFIELDS lines.

If $cols->item(0)->nodeValue gets you an error that means the HTML you got back was not in the format you thought it was.

Quote from: rcane on July 16, 2022, 12:50:48 PMIs it possible something in there is/isn't getting considered properly?
No. What's wrong here is that your code is being fed values and is doing things with them that you're not actually expecting; there is literally nothing wrong with SSI here except faulty cause and effect analysis.

Try printing the variables you're getting out before you try to use them to verify they are what you think they should be; I'm not sure this is actually true.

rcane

Quote from: Arantor on July 16, 2022, 12:59:45 PMOK, I'm deliberately going to not get into the fact that your SQL is doing some pretty hokey things it probably shouldn't ever be doing (why the MAX() against the themes table? That's... unique I guess), I suspect the bigger problem is that you're just making the cURL URL wrong.

I don't know if you know what this code is *actually* doing. It's probably not what you think it is, based on the comments you have in your code.

Your comment suggested that https://ziplook.house.gov/htbin/ziplook_find?773573016773573016 is a valid URL - and this produces a meaningful result if I go there.

If I construct the URL that you construct with that cURL, creating a POST that submits a form with separate parameters I just get an error.

It seems to me that what you need to do is create a URL of:
$url . $userzip . $userzip4 . $userzip . $user_thru
using your variables and change the cURL call to a GET rather than a POST which means dropping the CURLOPT_POST and CURLOPT_POSTFIELDS lines.

If $cols->item(0)->nodeValue gets you an error that means the HTML you got back was not in the format you thought it was.

Quote from: rcane on July 16, 2022, 12:50:48 PMIs it possible something in there is/isn't getting considered properly?
No. What's wrong here is that your code is being fed values and is doing things with them that you're not actually expecting; there is literally nothing wrong with SSI here except faulty cause and effect analysis.

Try printing the variables you're getting out before you try to use them to verify they are what you think they should be; I'm not sure this is actually true.

Ok,

1. I'll go back and employ your advice. Yes, I see it shouldn't be going the POST route
2. that meaningful url was just to test the site; it wasn't for any other purpose since I don't want to reproduce the .gov display but grab the values and use them in a format I see fit.
3. the code, however incorrectly assembled, produces the attached image results (which is what I was going for), but only if SSI.php isn't included/required.
4. the second image is how the results are if SSI "isn't" commented out. 

In the meantime, while I go through things: this what the curl does grab, as I had sought.  I'm not implying a problem with ssi to be causative with, but the two screenshots are at least correlative.

w/o ssi:
You cannot view this attachment.

with ssi:
You cannot view this attachment.

Arantor

Quote from: rcane on July 16, 2022, 01:17:01 PMI'm not implying a problem with ssi to be causative with

Yes, you are, actually.

Quote from: rcane on July 16, 2022, 01:17:01 PMthat meaningful url was just to test the site; it wasn't for any other purpose since I don't want to reproduce the .gov display but grab the values and use them in a format I see fit.

Well, it seems to me that it's the correct format to be getting data from since trying to reproduce the query that your cURL code creates doesn't work correctly when I try it.

In any case I still don't think that's the problem, nor do I think the code is doing what you think it's doing.

If you don't have a database connection, none of your variables get populated; the search or whatever you're calling with cURL will do something differently because presumably this time it doesn't have a partial ZIP code or whatever to filter on, meaning you get different results.

Which is why I suggested printing the variables out before they get to the cURL call so that you can verify what you're feeding into the cURL is what you think it should be because it probably isn't.

rcane

I printed the variables both when using the db and not. 
They were appropriately valued.

The choice of curl code being wrong aside, it's at least producing the expected results. But, I'll have to fix it with curl because I don't see a way to get the results by just tweaking the url. If "I" am going to leave out the zip4, it's got to be through the curl. I don't see how to write the url without the zip4 and NOT get what it will produce.

It's not meant to be offensive by mentioning ssi, but the original post WAS a question asking if it could interfere?   That's why I called it correlative. 

I've put the variables in manually (and printed) and used the db (and printed), and I've used a manual user id as well as with getting it via ssi. 

The data comes back, but for some reasons when ssi is referenced, it is as if the zip4 value gets wiped out and you get that second screen shot of many names.  But, the zip4 value isn't actually wiped out because I printed it before it did its thing.

Now, it might not be coming back from the curl (due to my bad code choices) so I have to go check that out.

But to make the analogy you never know what will happen if you spill coke on a motherboard, I cannot suspect beyond posting that, when I comment out ssi it works. 

I'm sure, as always, it's a gross over sight, but in the meantime it's not to tread on anyone's work.  Just trying to learn how things work.

rcane

Quote from: Arantor on July 16, 2022, 12:59:45 PMOK, I'm deliberately going to not get into the fact that your SQL is doing some pretty hokey things it probably shouldn't ever be doing (why the MAX() against the themes table? That's... unique I guess), I suspect the bigger problem is that you're just making the cURL URL wrong.

I had a difficult time getting a query to read down the variable table for a particular user and grab the value column 'value'.   Unique is one way of putting it.  If you have a more streamlined way I'm eager....

Arantor

I hope someone else can help you. There isn't actually enough information there for anyone to help you, but upon trying to push for the details that would help you just handwave them away as though you think they're not relevant.

rcane

Quote from: Arantor on July 16, 2022, 02:16:21 PMI hope someone else can help you. There isn't actually enough information there for anyone to help you, but upon trying to push for the details that would help you just handwave them away as though you think they're not relevant.

To which details are you referring?

Everything had been pasted.

live627

To further debug the code, print the result of the query and the built url

use var_dump() to see variables that are not strings.

rcane

Quote from: live627 on July 16, 2022, 11:04:42 PMTo further debug the code, print the result of the query and the built url

use var_dump() to see variables that are not strings.

The msql query was "unique" as Arantor put it as I had trouble getting values on the themes table on a previous task; i just recycled what had worked.  I have since refined that to something more preferred, but either way the results from the query are the same.

The nodevalue error is fixed as well.

This is the query and extracting the variables (only two per query are possible) to be used in the curl.

$gather = $conn->query(" SELECT value FROM smfqg_themes where id_member = $who AND variable IN('cust_zipcod', 'cust_zip4')");
  
         while($rows = $gather->fetch_object()) {
            $results[] = $rows;
            }
       
          $codes = json_decode( json_encode($results), true);
          $encode = json_encode($results); //
         
          foreach ($codes[0] as $var) { //get 5-digit zip
                $userzip = $var;
                }
          foreach ($codes[1] as $var) { //get the 4-digit zip
                $userzip4 = $var;
                }

These are the var_dumps:
var dump of $results
array(2) {
  • => object(stdClass)#37 (1) { ["value"]=> string(5) "77357" } [1]=> object(stdClass)#39 (1) { ["value"]=> string(4) "3325" } }


var dump of $codes
array(2) {
  • => array(1) { ["value"]=> string(5) "77357" } [1]=> array(1) { ["value"]=> string(4) "3325" } }


var dump of $encode
string(36) "[{"value":"77357"},{"value":"3325"}]"


These are the 'foreach' results: 

zips position 0 ($userzip) is: 77357 and is of type string
zips position 1 ($userzip4) is: 3325 and is of type string



rcane

The inspection of the form action (POST) @ https://ziplook.house.gov/zip/ziplook.html


You cannot view this attachment.


The build:
$fields = array(
            'zip' => $userzip,
            'zip4' => $userzip4,
            'thru_zip4' => $user_thru,
            );

$fields_string = http_build_query($fields);

echo $fields_string;

echo $fields_string yields:   zip=77357&zip4=3325&thru_zip4=3325







Advertisement: