Sticky

Testing challenge by James Bach 🚀

  • 26 June 2023
  • 28 replies
  • 3790 views

Userlevel 7
Badge +2
  • Community Manager
  • 103 replies

Attention ShiftSync community! We are excited to announce a brand new testing challenge prepared exclusively for you by the renowned expert, James Bach. The purpose of this challenge is to provide you with a way to showcase your analytical testing skills for other people to review and evaluate. You will get a winner badge and a certificate of achievement

Voting goes by likes. Like answers, which in your opinion provide the best solution to the challenge. The replies with the greatest number of likes will get winner badges.

Voting will be closed on 24.07.2023. However, even after the deadline, you may submit your solution in the comments and receive points.

 

Chapter

Here is a parking cost calculator for an airport:
https://developsense.com/parkcalc/index.php (This was copied from a real airport website, some years ago.)

There is a rumor that, because of poor input validation, you can make this calculator show huge parking costs (into the billions of dollars). Try to reproduce that. What is the highest amount you can make it quote you for parking?

Protocol

  • This is a hard problem meant to challenge you. There is no fixed time limit for solving it.
  • Use any resource you like to solve this problem. Tackling this with a team of people tends to work well.

Deliverable

  • Please explain your process in the comment section.
  • Take screenshots of your most interesting and extreme results and attach them to your comment.

 

About James Bach:
 

James Bach@satisfice  is a highly influential figure in software testing, renowned for his pioneering work on exploratory testing. His book, "Lessons Learned in Software Testing," co-authored with Cem Kaner and Bret Pettichord, is a go-to resource for testers worldwide. Bach emphasizes the value of skilled testing and encourages critical thinking, challenging traditional approaches. With his unwavering dedication, Bach has reshaped testing standards and inspired countless testers to think outside the box.

 

Learn more about James and his workSoftware Testing for Serious People - Satisfice, Inc.


28 replies

Userlevel 1

The sample parking calculator seems to be of an old application/website.

At first sight, it’s UI provides already some tips:

  • the OLD UI is a hot tip telling that it may not be responsive; which in fact, it’s not so it’s not going to work well with mobile devices and/or devices with different screen sizes
  • there seems to be several input that use the keyboard and also some shortcuts (e.g., for opening a calendar)

Focusing on the input validation, there are at least 2 ways of inputing data: keyboard and mouse. The fact that we can use both may present a challenge in terms of consistence between values edited using either of them.

Besides it, and probably more critical, is the fact that there seems to be no input validation whenever editing the fields using the keyboard; they are just text fields (i.e., “input” elements of “text” type), instead of proper HTML elements that could restrict it to just numbers or even a regular expression.

Knowing this, we can enter some invalid times like:

  • 15:00 AM (which is a value out of  bounds)
  • 15:00 PM (which becomes 27:00 AM after submitting)
  • 25:00 AM (which is a value out of  bounds)
  • -25:00 AM (which is a value out of  bounds)

More, 

  • the date field doesn’t have a validation, therefore we can enter a date in a whatever format. For example, in some countries it’s common to use YYYY-MM-DD, such as 2023/06/27. If we enter that for the end date, for example,  it will accept it, and returns a cost of more than 1,6 Million USD for a few days.

Additionally, we can see in the browser that submitting the values for calculation performs an HTTP GET, and the parameters can easily be modified on the browser address bar.

We can modify the type of Lot to some unexpected value like STPPP and use that to calculate the parking cost…  which won’t give an error and return $0, such in the following example.

 

https://developsense.com/parkcalc/index.php?Lot=STPPP&EntryTime=12%3A00&EntryTimeAMPM=AM&EntryDate=6%2F26%2F2023&ExitTime=12%3A00&ExitTimeAMPM=AM&ExitDate=6%2F27%2F2023&action=calculate&Submit=Calculate

 

 

More deep testing would be needed but the fact that there are some critical validations that aren’t being done, this parking calculator should not be released and we can even say that probably this needs to be fixed ASAP before more exhaustive testing is performed.

Userlevel 1

 

Having an adequate “Leaving Date and Time“ threshold would really help, e.g. 01/01/2100.
Also, please embed the link to the Parking Calc in the “Chapter” section, so that one doesn’t need to copy->paste the link or use “Open In New Tab OR Window”

Userlevel 6
Badge +2

Few descriptions are already provided on top.

The “Choose Leaving Date and Time” field accept real time data, with no restriction on decimal values, in fact it worked till 21 decimal values

 

result is here

 

Userlevel 3
Badge

I interacted around 45 minutes with the calculator straight and also during writing this report (around another 45 minutes).
Warning: I’m not used to AM/PM times therefore I might confuse some things which are normal for this.

My currently highest amount (It might even go higher): 5,777,922,043,452.00
 

Observations:

  • Dates are not checked for any sanity
  • A crash when I set the years date to high. The pages goes white after some seconds, having no UI anymore.
    • setting one of the dates to (-)350,000,000 let the site go white => It would need further investigations to find the exact limit.
    •  
  • Making a successful calculation with Valet Parking resets the option to Short-Term Parking
  • After a calculation with PM the UI is set back to AM and sometimes is the time also changed:
    • Warning: I’m not used to AM/PM times
    • 00:00 PM goes to 12:00 AM
    • 25:99 PM goes to 37:99 AM
  • Requesting the URL with working values returns the HTML immediately but it does not contains a cost value. With high, working dates takes the response some seconds.

How do I came to this?

At first I played a bit around to learn how this calculator works. What are the UI elements.

My first idea was that day dates in general and specifically extrem years are a fast way to create high amount.

I tried out if dates before 1.1.1970 (Unix date start) where an issue but even for that I could generate costs. Having the Leaving date before the Entry date was gave an error message as often as I tried it (maybe in combination with the time can be  found more issues).

Now I started playing around with the dates. At first going back in the Entry year like 999, -9. With the first -9 I just add 9 to reach the maximum which was -999,999,999 at that time, crashing the site.
As I observed with every 9 added the response took longer.

Now I set the Entry date to around -999 and tried in batches to approach the limit.

Maybe I should have stopped pushing the boundaries way earlier. Having Entry dates quite back in the past when no cars makes no sense at all. Also a Leaving date more than 100 years in the future is not that different from 1,000 years.

My time is out now. I have to stop the report here. Maybe I can continue and improve this report tomorrow.

Userlevel 1

I think I’ll leave out some of the details about the general overview of this calculator (i.e. the UI/UX shortcomings etc.) that have already been addressed here and jump straight into my findings.

First of all, it needs to be noted that the only input validation I’ve found in this calculator is the general check that the entry date shouldn’t be later than the exit date (and vice versa). That’s it. The inputs are not validated for non-digit characters which leaves plenty of room for bugs to crawl in (and hopefully be discovered).

So I started playing with numbers, particularly dates. My initial thought was to set the entry date as early as possible and the exit date as late as possible, hoping that the date difference would end up in a huge cost value returned. That was partially true, and I was able to get to something around a trillion. Adding up nines to the year section led to slower response time, so I decided to switch to time inputs.

It quickly became obvious that the the server did not have any restrictions on seconds part, and the number grew way quicker. 

That’s when I noticed that the number of days actually had scientific notation which was denoted by E+16 part. I wondered, well, if the server returns such a value to the client, then maybe it could process it if the client sends it to the server?
So I immediately tested that out by adding E+199 to the exit time which resulted in something like this…
 

As we can see, the cost number returned was so huge it didn’t even fit the page and broke the layout.
I went further and got curious what would happen if I entered an even bigger number in scientific notation, something around E+99999?
Below is the result

The return value turned out so huge that the calculator returned NAN dollars for infinite number of days.

That’s how far I’ve gone so far. Still wondering if there’s a way to make that thing return a negative value.

Userlevel 1

the calculator is using simple formula to calculate number of hours of parking and then multiplies it by a unit amount. The unit amount part looked ok to me

Time parked = (End Date End time - Start Date Start Time)

There is no validation on End Date End time or Start Date Start Time.

This is max amount I could get as below:

Start time: 0 Date : 0

End time:1.0E+304:1.0E+306

$28,888,888,888,888,888,216,804,184,759,942,138,746,047,761,733,137,639,962,131,879,827,712,316,167,361,170,965,208,303,495,542,144,880,552,591,661,274,497,301,372,395,918,746,356,108,582,811,181,673,959,222,080,850,792,644,042,856,288,782,018,504,299,610,380,302,930,039,590,665,547,643,580,580,615,527,391,425,986,093,822,742,404,520,595,613,706,021,388,377,148,501,889,585,833,721,752,900,338,167,316,480.00        (1.1111111111111E+303 Days, 0 Hours, 0 Minutes)

 

For any second above this, the message below is displayed


 

28,888,888,888,888,888,216,804,184,759,942,138,746,047,76

1,733,137,639,962,131,879,827,712,316,167,361,170,965,208,303,495,542,144,880,552,591,661,274,497,301,372,395,918,746,356,108,582,811,181,673,959,222,080,850,792,644,042,856,288,782,018,504,299,610,380,302,930,039,590,665,547,643,580,580,615,527,391,425,986,093,822,742,404,520,595,613,706,021,388,377,148,501,889,585,833,721,752,900,338,167,316,480.00        (1.1111111111111E+303 Days, 0 Hours, 0 Minutes)

Userlevel 4
Badge +4

It can be difficult to break out of our preconceived ideas of software limitations: for instance, the proper formatting of the date field.

This is what hackers do, though. This is why cybersecurity is so hard. Developers and product owners like to say “no user would do that.” But of course in cybersecurity terms, hackers are users, and they are motivated to find vulnerabilities in our products.

 

Userlevel 4
Badge +4

@Shrihari You are very close to the maximum I was able to get. My max is more than twice as large as yours, though.

Userlevel 4
Badge +4

@Sebastian Stautz For project purposes, I agree you don’t need to go any further. Still, it can be a good learning experience to try.

Userlevel 4
Badge +4

@darktelecom I like your report. It’s written clearly and seems well-reasoned. I agree that if we are simulating a realistic project situation, you don’t need to go any further. However, sometimes I like to push things beyond the practical on projects because I want to enhance my street cred as a bug investigator/hacker.

Userlevel 4
Badge +4

One tool I like to use is Hurl.

Here is the contents of a Hurl file you can use which might make it easier to systematically explore this problem:

GET https://developsense.com/parkcalc/index.php

[QueryStringParams]
Lot: STP
EntryTime: 8:00
EntryTimeAMPM: AM
EntryDate: 6/26/2023
ExitTime: 12:00
ExitTimeAMPM: PM
ExitDate: 6/26/2023
action: calculate
Submit: Calculate

HTTP 200
[Asserts]

xpath "string(/html/body/form/table/tr[4]/td[2])" contains "8.00"

You can keep updating the data and running Hurl. Or you can just add new sections to the file and keep all the history of your attempts.

Good for a non-functional tester 😅

 

Userlevel 2

I have chosen a simple combination of digits like 1 and 9 to check extremities randomly, manipulated the time data only and here is the result.

Why only a combination of 1s and 9s in the date fields?

Well, as a kid, we can think of only 1 and 9 to play with the smallest and largest values excluding 0.

And a wild attempt!

Tests: 

Here are my few random tests and after seeing this kind of profit, I would love to start a business.

  1. When all the inputs are in the wrong format.

     

  2. One Invalid date like 2/38/2100 is no barriers, and we are in profit.

     

  3.  Even negative signs did not matter!

     

  4. Following formats but with unexpected values and it works!

  5. No proper entries but the value of Zeros. Wondering how the calculation logic is working!

    Note: I am adding my observations later! Stay tuned.

Userlevel 4
Badge +4

Just a reminder: this software was real! It was taken from the Grand Rapids airport website, some years ago.

Userlevel 1

Please delete this comment if that's the case, but I guess "visiting in quick succession" 😉 this url https://developsense.com/parkcalc/index.php?Lot=STP&EntryTime=12%3A00&EntryTimeAMPM=AM&EntryDate=6%2F26%2F2023&ExitTime=12%3A00&ExitTimeAMPM=AM&ExitDate=6%2F26%2F1.0E%2B304%3A1.0E%2B306&action=calculate&Submit=Calculate

could make that backend sweat a little, huh?

Userlevel 4
Badge +4

@detestable I don’t get a reply at all with that input.

Userlevel 1

@satisfice ohhh, maybe it was just me, the request hanged for like 30 seconds and got me a blank response

 

bummer

 

 

Userlevel 1

@satisfice my mistake when reporting (tried to be the cool kid), doesn't work directly by url, but if you use 1.0E+304:1.0E+306 (obviously copied from Shrihari) as exit date and today as entry date and submit  I think you'll get it to hang

 

This was a fun little excercise. At first it was clear that some information is missing to fully test this. What are the supposed prices for every lot? Should I be able to stay for a Year on a Short Term Parking?

But then I though about the real issue I was supposed to investigate and decided to leave those questions for another time.

 

My first approach was to simply check how much could I adjust the dates. I tried to calculate a 30 year booking. Wich didn’t give me any unreasonable numbers. So I then decided to check if I could book a leaving date before the entry date. To my surprise this seemed to work quite nicely. So then I though about the time. (I also noticed that no error is displayed when the user invents dates like 29th of Febuary 2023)

 

Something that catched my eye is the AM/PM selector (I was thinking of inputing 24:00 AM). When I clicked the timers, I noticed I was able to write on them. So I started with the obvious 999999999999999999999 and the issue was found.

But I wasn’t done. I thought “Maybe if I am able to go that high I can go way down” so I decided to try with a big negative number “-99999999999999999” seemed reasonable.

 

Also, robert seems to be a valid date, and quite costly

 

 

No correct validation for displayed date format: possible days, months, years: min and max values. 

It looks like 00-99 range - when typed 2 symbols (DD, MM) and 0000-9999 range - 4 digits for YYYY. 

But as we know range for DD: 01-31; MM: 01-12; YYYY:  most possible start day - today. It means at least 2023 and till real year time (not 9999). But now you can type 0000 and it will be applied.

The maximum possible sum of money case:

Start date: 00/00/0000
End date: 99/99/9999

Sum: 76,041,838.00

 

Userlevel 5
Badge +3

Here is my report on this nice challenge.

 

My first goal was to find out how the app behaves with different date format(s) so I’ve tried the following formats: 

  • DD/MM/GGGG
  • GGGG/MM/DD
  • GGGG/DD/MM
  • YYYY-MM-DD
  • DD-MM-YYYY
  • MM-DD-YYYY
  • DD.MM.YYYY
  • MM.DD.YYYY
  • YYYY.MM.DD
  • YYYY.DD.MM
  • M.D.YYYY
  • D.M.YYYY
  • YYYY.M.D
  • YYYY.D.M
  • DD MM YYYY
  • MM DD YYYY
  • YYYY MM DD
  • YYYY DD MM
  • D M Y
  • D M YYYY
  • DD MM 'YY
  • DD-MM-YYYY HH:mm:ss
  • DD/MM/YYYY HH:mm:ss
  • DD.MM.YYYY HH:mm:ss

Next step was to find out how big value can ‘Date’ section accept before breaking and after some experimentation I came to the conclusion that the max/min values are in range, inclusive, -999999999 <> -999999999. 

 

The next step was to fiddle with the ‘Time’ inputs. There were some interesting finds here too. For example: 

  • Start date can be empty and will be set to ‘:00’ after calculation is performed

 

  • If ‘:XX’ is entered in the Leaving time and the ‘PM’ is selected the value will default to ‘12:XX’ after calculation

 

  • Setting start date to 00:00 AM and end date to 99:99 PM. After calculation, the 'end date' is set to '111:99'

 

  • The value in the 'Leaving date' increases with each calculation – 'PM' needs to be selected to achieve this (the pictures below show some of the results have ‘AM’ selected, that is because the picture was taken after the calculation was performed and the form resets some of the values after the calculation)

 

Then, there were some general issues like: 

  • The 'Choose a Lot' resets to 'Short-Term Parking' after each calculation

  • Entered values don’t reset after the page reload

 

 

Of course, the best one is saved for last. These are the two highest values I’ve managed to obtain (was inspired by results/inputs Shrihari obtained/used):

  • $12,708,333,333,333,332,967,401,980,232,859,168,628,525,464,503,698,613,414,522,724,605,015,219,418,325,420,468,736,234,296,598,399,533,039,132,395,113,466,895,373,157,943,356,627,796,442,973,405,410,397,248,800,629,166,261,831,448,178,493,452,611,580,311,562,459,296,227,824,353,005,193,212,658,209,101,865,408,410,243,734,079,892,283,725,736,687,007,252,967,632,114,030,003,305,119,796,382,192,088,645,632.00        (4.2361111111111E+302 Days, 0 Hours, 0 Minutes)

 

  • $24,999,999,999,999,998,481,338,812,638,411,554,650,100,718,050,293,312,382,976,928,928,308,011,407,533,084,757,108,273,143,601,269,371,092,140,295,140,405,431,446,792,984,356,590,076,325,589,497,102,167,206,937,468,253,604,205,027,602,669,275,632,906,102,264,609,299,506,371,378,997,691,599,206,377,054,581,648,872,780,674,019,874,513,365,087,296,656,431,016,019,010,952,114,899,655,187,260,870,345,359,360.00        (8.3333333333333E+302 Days, 2.7720004299093E+288 Hours, 0 Minutes)

Userlevel 3
Badge

@Sebastian Stautz For project purposes, I agree you don’t need to go any further. Still, it can be a good learning experience to try.

Somehow I found it more interesting to find bugs by playing with the UI than to look for the maximum cost :-)
Which might still be an interesting challenge on its own.

 

Userlevel 1

I did in the usual format I note down findings when testing (something I learned from RST). One can argue my “improvements” section doesn’t make sense but I seem them as something additional to the current scope. Everybody knows there’s a third party plugin out there somewhere that can do everything this calculator does.

 

Bugs: 

  • The "choose a lot" selection always resets to "Short-term Parking"

 

  • It takes negative values as input and also 0 as input

 

  • It takes time values greater that 24 as input

 

  • There's no way to refresh the thing since every parameter gets embedded in the URL

 

  • AM/PM should be decided based on time value selected. 1300 cannot be an AM value

 

  • Time value should auto format from 1200 to 12:00. Regex to detect minutes is missing, it takes the value entered as hours
  • The only way to get a time value in PM is to enter the AM equivalent, select PM radio button which then adds 12 hours to the value

 

  • If you input a PM value for time and then select the "PM" radio button, it will go beyond 2400

 

  • AM/PM radio buttons reset to AM
  • It even takes text as input for time and date

 

 

Improvements:

  • The date picker should be placed alongside the date field
  • The date picker should also include a time picker
  • There should be a max time limit to park your car
     
Userlevel 2

I have chosen a simple combination of digits like 1 and 9 to check extremities randomly, manipulated the time data only and here is the result.

Why only a combination of 1s and 9s in the date fields?

Well, as a kid, we can think of only 1 and 9 to play with the smallest and largest values excluding 0.

And a wild attempt!

Tests: 

Here are my few random tests and after seeing this kind of profit, I would love to start a business.

  1. When all the inputs are in the wrong format.

     

  2. One Invalid date like 2/38/2100 is no barriers, and we are in profit.

     

  3.  Even negative signs did not matter!

     

  4. Following formats but with unexpected values and it works!

  5. No proper entries but the value of Zeros. Wondering how the calculation logic is working!

    Note: I am adding my observations later! Stay tuned.

Observations:

What happens with a click on “Calculate”

  1. Choose any of the lots other than “Short-term Parking” from the dropdown, it will reset the value to the same.

Time Fields

  1. By standard norms, time follows the format of hh:mm.

I tried entering hh:mm:ss format and it got reset after clicking to calculate.

  1. Enter any time for the entry or leaving time field. (Default selecting is AM).

           Select the ‘PM’ button and calculate.

           You will get 12 hours added to ‘hh’.

           The only value that remains the same without change is 12:00.

           You can try any number from single zero to unlimited ranges.

  1. If you add only strings to the time field and calculate by selecting AM, it will show string:00
  2. If you add only strings to the time field and calculate by selecting PM, the value will reset to 12:00.
  3. If you add only time in hh:mm format followed by strings to the time field and calculate by selecting AM, it will remain the same.
  4. If you add time in hh format followed by strings to the time field and calculate by selecting AM, it will show the value as hhstring:00.
  5. If you add only time in hh:mm format followed by strings to the time field and calculate by selecting PM, it will show (hh+12):00string.
  6. If you add time in hh format followed by strings to the time field and calculate by selecting PM, it will remove the string, add 12 to hh and, set the value to (hh+12):00.
Userlevel 2

How the date field works

  1. No validation of date ranges, invalid date, invalid date format, date combined with strings, and special chars.
  2. If you simply enter just numbers (for easy fundamentals- pick anything between 1 to 12), it considers the number as the month like 1 stand for Jan, 2 for Feb, and so on, and calculates the total days of the entry month.

Note: If you select the entry date field with the value 0 and leave date field as 1, it will give you a default value of 31.

  1. If you enter the entry date in -/-/- format and 2nd field digit(s) is considered as day only.

The leave date if entered with a number between 1 and 12, is considered as month.

On calculation, the total days are counted as

{total number of the month (from leave date field)- days (from the entry date 2nd field)}

 

For the below example, total days = (31-12) = 19 days.

 

 

 

 

  1. If you enter both the entry and leave date in -/-/- format.

            On calculation, the total days are counted normally.

  1. No matter whatever value you add after -/-/- this, the values are counted based on the values entered within the -/-/- format.

Note: My other few points are already iterated by many of the others. 

Reply