Improvements to the payload definition | Community
Skip to main content

Improvements to the payload definition

oliverfarren
Forum|alt.badge.img

Hello,

Right now our typeform survey makes good use of the webhooks to send each response payload through our ETL backend for processing and analysis.

It’s been about six weeks and while we have things working, I have suggestions for improvements i’d like to see that would enable things to work better for us.

For us, typeform is a great way for the non-technical members of our team to have the ability to build a survey that looks good and quickly make changes from customer feedback.

However, as it stands we can run into issues where changes to the survey made from the admin create panel can easily break our ETL pipeline. We’d like to solve some of these problems and I have some suggestions that I think would help that I hope would not require a large amount of effort to implement.

Improvements to the payload

 

I want to discuss the relationship between the field objects and correspondent answer objects in the payload.

extract of the payload structure:

{
   "form_response": {
      "definition": {
         "fields": [
         ]
      },
      "answers": [
      ]
   }
}

Problem: multiple selection

Some questions, such as picture_choice, multiple_choice,  dropdown can have multiple answers selected. In the admin create panel, a range can be placed on the number of answers selected. However currently there is no way of knowing from the payload definition what this range is.

This is problematic for our backend payload validation since we right now we need to manually check the questions to get this information

Example:

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "picture_choice",
              "allow_multiple_selections": true,
              "ref": "animals_like",
              "properties": {},
              "choices": [
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Dog"
                },
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Cat"
                }
              ]
            } 
         ]
      },
      "answers": [
          {
            "type": "choices",
            "choices": {
              "labels": ["Dog"]
            },
            "field": {
              "id": "Z0W1B9ZJq45R",
              "type": "picture_choice",
              "ref": "animals_like"
            }
          },
      ]
   }
}

Solution: multiple selection

Add a range object to the properties:

"range": { 
   "minimum"
   "maximum"
}

 Example:

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "picture_choice",
              "allow_multiple_selections": true,
              "ref": "animals_like",
              "properties": {
                 "range": { 
                     "minimum": 1,
                     "maximum": 2
                  }
              },
              "choices": [
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Dog"
                },
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Cat"
                }
              ]
            } 
         ]
      }
}

Having a range object for multiple selections AND opinion scales, other numerical questions would be very useful in allowing us to validate the payload from just the definition.

Problem: dropdown questions have no “choices” object

The same picture choice question above can be easily translated to a dropdown question with the labels being preserved. However for some reason there is no “choices” object as part of the definition when a question is dropdown? This seems like it shouldn’t be the case.

Having the “choices” object in the definition is very useful for us to validate if any changes have been made to the survey between payloads. By not having this in the definition of a dropdown question it can make our pipeline open to error since we might have a bug that is only caught when a bad answer is received. Or otherwise, someone has to manually check each answer label for each dropdown question in the admin panel which is in itself error prone.

Example:

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "dropdown",
              "allow_multiple_selections": true,
              "ref": "animals_like",
              "properties": {}
         ]
      },
      "answers": [
          {
            "type": "choices",
            "choices": {
              "labels": ["Dog"]
            },
            "field": {
              "id": "Z0W1B9ZJq45R",
              "type": "dropdown",
              "ref": "animals_like"
            }
          },
      ]
   }
}

Solution: dropdown questions have no “choices” object

Should be instead:

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "dropdown",
              "allow_multiple_selections": true,
              "ref": "animals_like",
              "properties": {},
              "choices": [
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Dog"
                },
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Cat"
                }
              ]
            } 
         ]
      },
      "answers": [
          {
            "type": "choices",
            "choices": {
              "labels": ["Dog"]
            },
            "field": {
              "id": "Z0W1B9ZJq45R",
              "type": "dropdown",
              "ref": "animals_like"
            }
          },
      ]
   }
}

Problem: We don’t know which questions are required

 

In the admin create panel there is a switch to make a question “required”. We would like to have a field added to the definition that reflects this information so we know whether to always or only sometimes expect an answer for a given question. This would allow us to validate whether we are getting answers when we expect too.

Solution: We don’t know which questions are required

Add a boolean field to the definition

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "dropdown",
              "allow_multiple_selections": true,
              "required": true,
              "ref": "animals_like",
              "properties": {}
            }
         ]
      }
   }
}

Generating a better test request

 

Our survey is currently hooked to an endpoint so each payload can be processed.

Right now there are only two ways to test any changes to our survey and each has drawbacks.

  1. Manually go through our survey and generate a test payload
  2. In the admin/connect/webhooks panel click the test request button

I really like the idea of option 2, I’d like it if I could click the test request button and if it comes back with a response of 200 then I should be able to conclude there are no breaking changes that have been made by our marketing team. It’s a great smoke test with minimal effort.

However the problem I have is that the test request that’s generated is not good.

Right now, every answer is “Barcelona”

{
   "form_response": {
      "definition": {
         "fields": [
            {
              "id": "Z0W1B9ZJq45R",
              "title": "What animals do you like?",
              "type": "picture_choice",
              "allow_multiple_selections": true,
              "ref": "animals_like",
              "properties": {},
              "choices": [
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Dog"
                },
                {
                  "id": "nqHeZVBfCt2L",
                  "label": "Cat"
                }
              ]
            } 
         ]
      },
      "answers": [
          {
            "type": "choices",
            "choices": {
              "labels": ["Barcelona"]
            },
            "field": {
              "id": "Z0W1B9ZJq45R",
              "type": "picture_choice",
              "ref": "animals_like"
            }
          },
      ]
   }
}

Because we have validation checks, this doesn’t allow us to use this option.

I would like it if this could be improved so that a valid response could be generated. I believe if the definition object of the payload were improved by my suggestions above, it would be all the sufficient information required to generate a valid set of answers and this would provide immense value to us as a team.

API call to generate a test request

Further still to generating a test request through either the admin panel or connect panel, I would like it if I could send a GET request to return a test payload.

This would be very useful since we could then build an automated process to monitor for changes and be proactive to problems.

 

Thanks for reading, I hope we can discuss my problems and suggestions and colloborate on solutions.

 

Best regards,

Oliver

 

Collecting data in style.

4 replies

oliverfarren
Forum|alt.badge.img
  • oliverfarren
  • Author
  • Socializer
  • 9 replies
  • June 16, 2021

Additionally,

I notice that the “allow_multiple_selections”  field is either always true, or not present. It would be preferable if this field were always present and the value not always true.

Collecting data in style.
picsoung
Typeform
Forum|alt.badge.img+5
  • picsoung
  • Developer Advocate @ Typeform
  • 390 replies
  • June 22, 2021

Thank you Oliver for such a detailed message full of great suggestions!
Great to see you are using Webhooks to connect it to your ETL solution, would love to know which one you are using, maybe it’s worth spending time building a plugin/integration for this ETL if more people ask for it.

Most of your comments are related to the webhook payload missing details. The best approach is probably to rely on the form definition endpoint (GET https://api.typeform.com/forms/{form_id}) as it’s more complete.
We had to make a comprise between a verbose payload and something useful.


But you are totally right, you should expect the whole definition here.

The range you mention exists for multiple_choice fields, but it’s not available in the webhook payload.

"validations": {
        "required": false,
        "min_selection": 1,
        "max_selection": 3
      }


On the test request, we focus on giving you something that would have more or less the same shape for the payload, giving you placeholder for data.

I understand that the placeholder data is not really helpful or confusing. This is probably something we could improve on multiple_choice, picture_choice, dropdown and other fields, so it sends real choices. Until then I would recommend to fill up the form once, and resend this payload as many times as you need to debug your handler.

Hopefully this helps :)
Please keep the feedback coming!

john.desborough
oliverfarren
T
3 people like this
  • john.desborough
    john.desborough
  • oliverfarren
    oliverfarren
  • T
    TMo
Gabriel
Ex–Typefomer
Forum|alt.badge.img+5
  • Gabriel
  • Community Team
  • 857 replies
  • June 22, 2021

Let me thank you too, @oliverfarren, for some really great suggestions. I shared this yesterday with our team internally and the first reaction I got was ‘WOW”. :bow_tone4:

"If you think you are too small to make a difference, try sleeping with a mosquito".
john.desborough
oliverfarren
2 people like this
  • john.desborough
    john.desborough
  • oliverfarren
    oliverfarren
oliverfarren
Forum|alt.badge.img
  • oliverfarren
  • Author
  • Socializer
  • 9 replies
  • June 22, 2021

Thankyou for your responses @picsoung and @Gabriel.

@picsoung, regarding the ETL solution, currently the webook is connected to an AWS lambda layer in which we have a custom python package that transforms the payload and writes to a database. So perhaps not what many others are doing and why I might have experienced more issues since I am working directly with the payload.

I hadn’t considered the Create API as a method of monitoring for changes to the form definition. I’ll have a play with this some more and see whether this could be used as a workaround to solve some of the problems we were having. On a brief look it looks like it could solve a lot of problems, thanks!

I appreciate the compromise between verbosity and utility. Though in which case I wonder whether having the option to customise what gets sent in the payload might bridge that and consolidate development of a single source of truth definition and be a real value add for users like myself or those designing integrations?

Though following that thought further, comparing the response payload definition to the GET api’s definition I notice they’re very similar in structure and at first I thought one must be a subset of the other, but now I see they deviate somewhat.

So in the Create API I see a structure like:

{
   "fields":[
      {
          "title":,
          "ref":,
          "properties":{
              "allow_multiple_selection": false,
              "choices": [...]
          },
          "type":
      }
   ]
}

Whereas in the webhook payload the structure is like:

{
   "fields":[
      {
          "title":,
          "ref":,
          "allow_multiple_selections":true,
          "properties":{},
          "choices": [...],
          "type":
      }
   ]
}

I don’t want to be overly critical since I acknowledge I’m not in a position to appreciate the design decisions or evolution but from my own experience as a user i’m curious whether such a design is more prone to inconsistency - such as the webhook dropdown question not having the choice object list. In addition to making the maintainability more challenging since any change or extension to the payload definition must presumably now be updated in multiple places for both the GET api definition AND the webhook to work.

Thanks for listening to my feedback.

Best regards,

Oliver

 

 

 

 

Collecting data in style.
john.desborough
1 person likes this
  • john.desborough
    john.desborough

Reply


Still need help?

Don't be shy and ask the community.

Ask your question here!
Community T&Cs