One of the padded_string examples gives run-time failure #2285

pmuvb · 2024-11-01T15:42:30Z

pmuvb
Nov 1, 2024

Here is the code:

// clang++ -std=c++23  -g3 -O0 -W'all' -W'extra' -pedantic simdjson.cpp padded04.cpp 

#include "simdjson.h"
#include <iostream>
using namespace simdjson; // optional

int main(void) 
{
    // You can copy your data directly on a simdjson::padded_string as follows:
    const char * data = "my data"; // 7 bytes
    simdjson::padded_string my_padded_data(data, 7); // copies to a padded buffer
    std::cout << "strlen=" << strlen(data) << ", sizeof=" << sizeof(data) << "." << std::endl;
    ondemand::parser parser;
    ondemand::document doc = parser.iterate(data, strlen(data), sizeof(data));
    return 0;
}

Here is the output:
strlen=7, sizeof=8.
terminate called after throwing an instance of 'simdjson::simdjson_error'
what(): INSUFFICIENT_PADDING: simdjson requires the input JSON string to have at least SIMDJSON_PADDING extra bytes allocated, beyond the string's length. Consider using the simdjson::padded_string class if needed.
Aborted

Just downloaded simdjson 3.10.1.

The sizeof() should be 72 bytes.

Why is the error?

Tx

pmuvb · 2024-11-01T15:45:28Z

pmuvb
Nov 1, 2024
Author

If I use this iterate:

ondemand::document doc = parser.iterate(data);

it gives the same error.

0 replies

lemire · 2024-11-01T16:43:16Z

lemire
Nov 1, 2024
Maintainer

One of the padded_string examples gives run-time failure

The sizeof() should be 72 bytes.

This is unrelated to simdjson. You misunderstand the sizeof operator in C/C++. Given any pointer, sizeof(mypointer) will give you, typically, 8 bytes on a 64-bit system.

It does not do what you think it does.

You should be able to do...

const char * data = "my data"; 
simdjson::padded_string my_padded_data(data, 7);
ondemand::parser parser;
ondemand::document doc = parser.iterate(my_padded_data);

Give it a try!

1 reply

pmuvb Nov 1, 2024
Author

My thinking was padded_string expands (somehow) the pointer into array. I missed that I needed to use my_padded_data not data.

K, it works,

TX

pmuvb · 2024-11-01T17:41:52Z

pmuvb
Nov 1, 2024
Author

Since we are on a topic, why I cannot handle square brackets?

// clang++ -std=c++23  -g3 -O0 -W'all' -W'extra' -pedantic simdjson.cpp padded07.cpp 


#include "simdjson.h"
#include <iostream>
using namespace simdjson; // optional


int main(void) 
{
    ondemand::parser parser;
    char json[1024 + simdjson::SIMDJSON_PADDING] = R"([{"sym":"AAPL","vw":223.5643}])";
//        char json[1024 + simdjson::SIMDJSON_PADDING] = R"({"sym":"AAPL","vw":223.5643})";
    ondemand::document doc = parser.iterate(json, strlen(json), sizeof(json));
    std::cout << std::string_view(doc["sym"]) << ", " << std::endl;

    return 0;
}

produces:
terminate called after throwing an instance of 'simdjson::simdjson_error'
what(): INCORRECT_TYPE: The JSON element does not have the requested type.
Aborted

Square brackets are for the arrays, but an array can contain just a single element.

0 replies

pmuvb · 2024-11-01T17:47:44Z

pmuvb
Nov 1, 2024
Author

nm,

I need to iterate using object.

0 replies

lemire · 2024-11-01T17:51:58Z

lemire
Nov 1, 2024
Maintainer

@pmuvb Your JSON document is

[{"sym":"AAPL","vw":223.5643}]

It is an array with just one element in it.

You are trying to do doc["sym"]. This cannot work since doc is an array.

Are you sure you do not mean...

#include "simdjson.h"
#include <iostream>
using namespace simdjson;

int main(void) {
  ondemand::parser parser;
  auto json = R"({"sym":"AAPL","vw":223.5643})"_padded;
std::cout << json << std::endl;
  ondemand::document doc = parser.iterate(json);
  std::cout << std::string_view(doc["sym"]) << std::endl;
  return 0;
}

I suspect that your difficulty is with the C++ syntax here.

0 replies

pmuvb · 2024-11-01T17:57:03Z

pmuvb
Nov 1, 2024
Author

That's how I receive the data, sometimes they contain just one element sometimes multiple, but vendor is always using square brackets:

[{"ev":"A","sym":"NVDA","v":110,"av":1028901,"vw":135.5109,"o":135.51,"c":135.51,"h":135.51,"l":135.51,"a":135.245,"z":55,"s":1730457518000,"e":1730457519000}]

but, yes, I know how to handle it:

// clang++ -std=c++23  -g3 -O0 -W'all' -W'extra' -pedantic simdjson.cpp padded07.cpp 


#include "simdjson.h"
#include <iostream>
using namespace simdjson; // optional


int main(void) 
{
    ondemand::parser parser;
    char json[1024 + simdjson::SIMDJSON_PADDING] = R"([{"sym":"AAPL","vw":223.5643}])";
    for (ondemand::object el : parser.iterate(json, strlen(json), sizeof(json))) {
        std::cout << std::string_view(el["sym"]) << ", " << el["vw"].get_double() << std::endl;
    }
    return 0;
}

TX for responding

I don't know why I am having trouble with code formatting here, it doesn't do it for me.

0 replies

lemire · 2024-11-01T18:02:06Z

lemire
Nov 1, 2024
Maintainer

@pmuvb You can also use JSON Pointer expressions if you really just need to pick a few bits of information, such as in this example...

#include "simdjson.h"
#include <iostream>
using namespace simdjson;

int main(void) {
  ondemand::parser parser;
  auto json = R"([{"sym":"AAPL","vw":223.5643}])"_padded;
  ondemand::document doc = parser.iterate(json);
  std::cout << doc.at_pointer("/0/sym") << std::endl;
  return 0;
}

This will not be performant if you do this repeatedly, however. The loop you are doing is the proper way to handle it.

0 replies

pmuvb · 2024-11-01T18:10:43Z

pmuvb
Nov 1, 2024
Author

These are multiple elements in array:

[{"ev":"A","sym":"FTV","v":100,"av":1861991,"op":71.4,"vw":71.26,"o":71.26,"c":71.26,"h":71.26,"l":71.26,"a":71.7974,"z":100,"s":1730483301000,"e":1730483302000},{"ev":"A","sym":"VMD","v":229,"av":18870,"op":8.61,"vw":8.6286,"o":8.63,"c":8.63,"h":8.63,"l":8.63,"a":8.6591,"z":57,"s":1730483301000,"e":1730483302000},{"ev":"A","sym":"BBIO","v":100,"av":924557,"op":23.51,"vw":25.21,"o":25.21,"c":25.21,"h":25.21,"l":25.21,"a":24.8885,"z":100,"s":1730483301000,"e":1730483302000}]

Right, I need to read those in a loop, hundreds, if not thousands, in a second. Hence, performance is a concern, I will be parsing all those values into a structure.

1 reply

lemire Nov 1, 2024
Maintainer

Right, I need to read those in a loop, hundreds, if not thousands, in a second. Hence, performance is a concern, I will be parsing all those values into a structure.

A loop is efficient.

If you encounter performance issues, just construct a reproducible test case and submit it to us.

pmuvb · 2024-11-04T19:52:17Z

pmuvb
Nov 4, 2024
Author

Since offered, could you take a look at my parsing code for a quick comment))
It just reads JSON messages from a websocket inside an infinite loop and de-serializes them into a structure.

What to look:

performance is a concern.
error handling. that infinite while(true) must keep going.

I did test it for error handling, it works.

build: $ clang++ -std=c++23 -g3 -O0 -W'all' -W'extra' -pedantic -DSIMDJSON_DEVELOPMENT_CHECKS=1

// The size of this buffer determines how much data I can read from a websocket at a time.
size_t const MESSAGE_BUFFER_SIZE = 2048 + simdjson::SIMDJSON_PADDING;


// Contains polygon JSON message.
struct Message {
    char buffer[MESSAGE_BUFFER_SIZE] = {'\0'};
};


void poly_parse(std::function<bool(Message& value)> callback_pop_from_polyfh, std::function<bool(Bar& value)> 
    callback_try_push_to_analytics)
{
    Bar bar;
    Message message;
    size_t sz = sizeof(message.buffer);
    simdjson::ondemand::parser parser;
    simdjson::ondemand::document doc;
    simdjson::ondemand::array json_bars; 
    while(true) {
        {
            bool const rc = callback_pop_from_polyfh(message);
            if (false == rc) {
                log_warning("Failure in polyparser to pop a message from polyfh, continue.");
                continue;
            }
        }

        {
            size_t len = strlen(message.buffer);
            auto const rc = parser.iterate(message.buffer, len, sz).get(doc);
            if (simdjson::SUCCESS != rc) { 
                log_error("Failure to acquire document due to rc=\"%s\", continue.", simdjson::error_message(rc));
                continue;
            }
        }

        {
            auto const rc = doc.get_array().get(json_bars);
            if (simdjson::SUCCESS != rc) { 
                log_error("Failure to acquire array due to rc=\"%s\", continue.", simdjson::error_message(rc));
                continue;
            }
        }

        for (auto json_bar : json_bars) {
            simdjson::ondemand::object obj;
            auto const rc = json_bar.get_object().get(obj);
            if (simdjson::SUCCESS != rc) { 
                log_error("Failure to acquire object due to rc=\"%s\", continue.", simdjson::error_message(rc));
                continue; 
            }

            int const rc0 = poly_parse_object(bar, obj);
            if (1 == rc0) {
                log_warning("Failure in polyparser to parse an object=\"%s\", continue.", message.buffer);
                continue;
            }

            bool const rc1 = callback_try_push_to_analytics(bar);
            if (false == rc1) {
                log_warning("Failure in polyparser to try-push to analytics bc buffer is full, continue.");
            }
        }
    }
}


int poly_parse_object(Bar& bar, simdjson::ondemand::object& obj)
{
    {
        std::string_view value;
        auto const rc = obj["sym"].get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing symbol due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        size_t len = value.length(); 
        std::strncpy(bar.symbol, value.data(), len);
        bar.symbol[len] = '\0';
    }

    {
        long value = 0;
        auto const rc = obj["v"].get_int64().get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing bar volume due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        bar.volume = value;
    }

    {
        long value = 0;
        auto const rc = obj["av"].get_int64().get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing today's volume due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        bar.today_volume = value;
    }

    {
        double value = 0;
        auto const rc = obj["o"].get_double().get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing open due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        bar.open = static_cast<long>(value * POLYGON_PRICE_MULTIPLIER);
    }

    {
        double value = 0;
        auto const rc = obj["c"].get_double().get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing close due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        bar.close = static_cast<long>(value * POLYGON_PRICE_MULTIPLIER);
    }

    {
        long value = 0;
        auto const rc = obj["s"].get_int64().get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing start timestamp due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        bar.ts = value;
    }

    return 0;
}

1 reply

lemire Nov 4, 2024
Maintainer

In general, something like size_t len = strlen(message.buffer); requires parsing the whole of the data to find its length. Is this necessary? If possible, I recommend working with std::string_view instances.

Generally, your code can be simplified. E.g. you should be able to just do the following:

     if(auto rc = obj["av"].get_int64().get(bar.today_volume); rc) {
            log_error("Failure parsing today's volume due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
     }

And so forth.

The following code is unfortunate. You almost certainly do not want to use strncpy. A simple memcpy is also surely want you want.

        std::string_view value;
        auto const rc = obj["sym"].get(value);
        if (simdjson::SUCCESS != rc) {
            log_error("Failure parsing symbol due to rc=\"%s\", return.", simdjson::error_message(rc));
            return 1;
        }
        size_t len = value.length(); 
        std::strncpy(bar.symbol, value.data(), len);
        bar.symbol[len] = '\0';

I trust you know that you have enough memory allocated in bar.symbol?

The key order should match as much as possible those expected in your document ("sym", "v", etc.) for best performance.

pmuvb · 2024-11-09T18:15:40Z

pmuvb
Nov 9, 2024
Author

Got rid of the strlen and replaced strncpy with memcpy, simplified retrieving the values.

I believe I got the most out of the lib.

I appreciate your code review!

0 replies

One of the padded_string examples gives run-time failure #2285

Uh oh!

Uh oh!

pmuvb Nov 1, 2024

Replies: 10 comments · 3 replies

Uh oh!

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

lemire Nov 1, 2024 Maintainer

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

lemire Nov 1, 2024 Maintainer

Uh oh!

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

lemire Nov 1, 2024 Maintainer

Uh oh!

Uh oh!

pmuvb Nov 1, 2024 Author

Uh oh!

lemire Nov 1, 2024 Maintainer

Uh oh!

Uh oh!

pmuvb Nov 4, 2024 Author

Uh oh!

Uh oh!

lemire Nov 4, 2024 Maintainer

Uh oh!

Uh oh!

pmuvb Nov 9, 2024 Author

pmuvb
Nov 1, 2024

Replies: 10 comments 3 replies

pmuvb
Nov 1, 2024
Author

lemire
Nov 1, 2024
Maintainer

pmuvb Nov 1, 2024
Author

pmuvb
Nov 1, 2024
Author

pmuvb
Nov 1, 2024
Author

lemire
Nov 1, 2024
Maintainer

pmuvb
Nov 1, 2024
Author

lemire
Nov 1, 2024
Maintainer

pmuvb
Nov 1, 2024
Author

lemire Nov 1, 2024
Maintainer

pmuvb
Nov 4, 2024
Author

lemire Nov 4, 2024
Maintainer

pmuvb
Nov 9, 2024
Author