SID-02381: Invalid crypt token when using CGI scripts to update topic

Status:	Answered	TWiki version:	6.0.1	Perl version:
Category:	CategoryAPI	Server OS:		Last update:	6 years ago

I'm trying to update a Twiki topic by making requests on TWiki's CGI scripts. The following GET works perfectly:

def get_topic(topic_path):
  url       = "https://mytwiki.comtwiki/"
  twiki_cgi = "{:s}/bin/view/{:s}".format(url,topic_path)
  auth      = ("username", "password")

  params    = {'username': 'username', 'password': 'password 'raw': 'text'}
  response  = requests.get(twiki_cgi, auth=auth, verify=False, params=params)

  return response

while using the save script (with a POST request), yields an invalid crypt token error page.

def set_topic(topic_path):
  url       = "https://my_twiki.comtwiki/"
  twiki_cgi = "{:s}/bin/save/{:s}".format(url,topic_path)
  auth      = ("username", "password")

  with open("new_topic_text.twiki", 'r') as read_file:
    topic_text = read_file.read()
  
  data      = {'username': 'username 'password': 'password', 'text': topic_text}
  response  = requests.post(twiki_cgi, auth=auth, verify=False, params=params, data=data)

  return response

Is there a way to not have that error when using save from something else than a browser?

-- TWiki Guest - 2018-11-29

Discussion and Answer

TWiki uses a crypt token on save for CSRF/XSS protection. Two options:

1. Disable it in configure by setting {CryptToken}{Enable} to false. Not recommended on a public site.

2. Provide a valid crypt token to the save script. To get that token, simulate an edit as an authenticated user and grab the value of the hidden field named crypttoken - it will look like this: <input type="hidden" name="crypttoken" value="e46b809bfeebfbec9c45324f4ed1f6cb" />. Supply that crypt token value as a parameter to the save script, it is only for one time use.

-- Peter Thoeny - 2018-11-29

If I understand correctly, I would need to simulate an edit from the web interface, by actually clicking on the edit button, and then supply that to my script?

In other words, to interact exclusively programmatically with twiki, I would need to disable the CryptToken?

-- TWiki Guest - 2018-12-03

No, use first requests.get() on the edit script, programmatically extract the crypt token value, then use requests.get() again to save your data. The former one can be on any page, it's just used to get the crypt token.

-- Peter Thoeny - 2018-12-03

Ok ok, perfect, I'll try that out. Thanks!

-- TWiki Guest - 2018-12-04

This also yields an invalid crypt token.

def set_topic(topic_path):
 # -- Edit page to get crypttoken.
 url = "https://mytwiki.com/twiki/"
 twiki_cgi = "{:s}/bin/edit/{:s}".format(url,topic_path)
 auth = ("username", password)
 params = {'username': 'username', 'password': password}

 response = requests.get(twiki_cgi, auth=auth,verify=False,params=params)

 # -- Parse the HTML to get the crypttoken value.
 soup = BeautifulSoup(response.text, 'html.parser')
 crypttoken = soup.find(attrs={"name": "crypttoken"})['value']
 params['crypttoken'] = crypttoken

 with open("response.html", 'r') as read_file:
  topic_text = read_file.read()

 # -- Save new topic with new text and crypttoken taken from earlier edit.
 twiki_cgi = "{:s}/bin/save/{:s}".format(url,topic_path)
 data = {'username': 'username', 'password': password, 'text': topic_text, 'crypttoken': crypttoken}

 response = requests.post(twiki_cgi, auth=auth, verify=False, data=data, params=params)

 return response

I've tried adding the crypttoken to both data and params, only data, only params, all yield invalid crypt token.

-- TWiki Guest - 2018-12-05

I don't know Python so I can't follow what you did. What is the crypttoken value you get? It should be a long hex number, such as 0782bb3df95259d705cf28990c39d678. It can be used only once. Adding it into the post data as key crypttoken should work.

-- Peter Thoeny - 2018-12-06

The first paragraph, until response = requests.get(...), queries the edit script. With Beautiful Soup, I parse the resulting response for the crypttoken, which is indeed a long hex number (something like 712175228f2b64422f8d07ad96a5a26e). I then pass crypttoken in the POST request with data=data. It still gives an invalid crypt token error.

-- TWiki Guest - 2018-12-06

Ok, I seem to be having an issue on my end. Printing the raw requests shows that both requests.get and requests.post give GET requests.

-- TWiki Guest - 2018-12-06

Figured it out.

When I removed the params argument of requests.post, I got redirected to the login page. Reinstating it yielded the invalid crypt token message. The solution was to use a Session object to maintain the login cookie so as to not get redirected to the login page. So, this works:

<verbatim> def set_topic(topic_path): # -- Create a session. url = "https://rcsgwiki.ext.nrc.ca/twiki/" auth = ("username", password) params = {'username': 'username', 'password': password}

s = requests.Session() s.auth = auth

# -- Grab the crypttoken by editing the page but doing nothing. twiki_cgi = "{:s}/bin/edit/{:s}".format(url,topic_path) response = s.get(twiki_cgi,verify=False,params=params)

# -- Parse the HTML to get the crypttoken value. soup = BeautifulSoup(response.text, 'html.parser') crypttoken = soup.find(attrs={"name": "crypttoken"})['value'] params['crypttoken'] = crypttoken

with open("response.html", 'r') as read_file: topic_text = read_file.read()

twiki_cgi = "{:s}/bin/save/{:s}".format(url,topic_path) data = {'username': 'username', 'password': password, 'text': topic_text, 'crypttoken': crypttoken} response = s.post(twiki_cgi, data=data,auth=auth,verify=False)

for r in response.history: print(r.status_code,r.url)

return response </verbatim>

-- TWiki Guest - 2018-12-06

I'm having the same error.

Checking the edit page source, I found out that TWiki is not generating any hidden field 'crypttoken' within the 'save' form. And this appears to happen only for this page.

Could this be caused by the content of the page?

I already tried the following;

• logout/login
• disable/enable crypttoken in the configure page
• delete cgi_session files under working/tmp
• copy the content to a new Topic

but none worked.

Any hint? Any Perl log that I can check anywhere?

Many thanks, Thomas

-- Thomas Fozzi - 2019-09-09

If you answer a question - or someone answered one of your questions - please remember to edit the page and set the status to answered. The status selector is below the edit box.