Set LOGFAIL= one of ALL, DEBUG, INFO, WARN, ERROR, NONE. A passing test will
run silently, as now; but a failing test will replay log output at the
specified level or higher.
While at it, support LOGTEST environment variable, same values. This is like
setting --debug (or -d), but allows specifying an arbitrary level -- and,
unlike --debug, can be set for a TeamCity build config without modifying any
scripts or code.
Publish LLError::decodeLevel(std::string), previously private to llerror.cpp.
That lets us reliably declare the operator<<() free function inline, which
permits multiple translation units in the same executable to #include
"wrapllerrs.h".
While we're accumulating the 'length:' prefix, the present socket-based logic
reads 20 characters, then reads 'length' more, then discards any excess (in
case the whole 'length:data' packet ends up being less than 20 characters).
That's probably a bug: whatever characters follow that packet, however short
it may be, are probably the 'length:' prefix of the next packet. We probably
only get away with it because we probably never send packets that short.
Earlier llleap_test.cpp plugin logic still read 20 characters, then, if there
were any left after the present packet, cached them as the start of the next
packet. This is probably more correct, but complicated. Easier just to read
individual characters until we've seen 'length:', then try for exactly the
specified length over however many reads that requires.
In load testing, we have observed intermittent failures on Windows in which
LLSDNotationStreamer into std::ostringstream seems to bump into a hard limit
of 1048590 bytes. ostringstream reports that much buffered data and returns
that much -- even though, on examination, the notation-serialized stream is
incomplete at that point. It's our intention to load-test LLLeap and
LLProcess, not the local iostream implementation; we hope that this kind of
data volume is comfortably greater than actual usage. Back off the
load-testing max size a bit.
On Windows we ran into trouble trying to write a biggish (~1 MB) buffer of
data to the child process's stdin pipe with a single apr_file_write() call.
The child actually received corrupted data -- suggesting a possible bug in
either APR or Windows pipes; the same test driving the same logic worked fine
on Mac and Linux. Empirically, iterating over chunks of the buffered data is
more robust.
New llleap_test.cpp load testing turned up Windows issue in which plugin
process received corrupt packet, producing LLSDParseError. Add code to dump
the bad packet in that case -- but if LLSDParseError is willing to state the
offset of the problem, not ALL of the packet.
Quiet MSVC warning about little internal base class needing virtual destructor.
Previous "read N of M bytes" wording implied that the child had M bytes to
send, but we only read N of them. In reality we have no idea how many bytes
the child is trying to send, only how many the OS is willing to deliver at
this moment. To me, "filled N of M bytes" more clearly implies that M is the
buffer size.
It only took a few examples of trying to wrangle notation LLSD as string data
to illustrate how clumsy that is. I'd forgotten that a couple other TUT tests
already invoke Python code that depends on the llsd module. The trick is to
recognize that at least as of now, there's still an obsolete version of the
module in the viewer's own source tree. Python code is careful to try
importing llbase.llsd before indra.base.llsd, so that if/when we finally do
clear indra/lib/python from the viewer repo, we need only require that llbase
be installed on every build machine.
Migrate logic from specific test to common reader module, notably parsing the
wakeup message containing the reply-pump name.
Make test script post to Result struct to communicate success/failure to C++
TUT test, rather than just writing to log.
Make test script insensitive to key order in serialized LLSD::Map.
Instantiating LLLeap with a command to execute a particular child process sets
up machinery to speak LLSD Event API Plugin protocol with that child process.
LLLeap is an LLInstanceTracker subclass, so the code that instantiates need
not hold the pointer. LLLeap monitors child-process termination and deletes
itself when done.
This arises, for instance, if you want to be able to create a temporary Python
module you can import from test scripts. The Python module file MUST have the
.py extension.
All known callers were using ensure(! withMessage(...).empty()). Centralize
that logic. Make failure message report the string being sought and the log
messages in which it wasn't found.
In case someone does want to permit the search to fail, add an optional
'required' parameter, default true.
Leverage new functionality in llprocess_test.cpp.
We were using uniform macro to report the APR function and its C++ parameter
expressions. But specifically for apr_proc_create() failure, better to report
the command we're attempting to execute.
Giving more unit tests the ability to capture and examine log output is
generally useful. Renaming the class just makes it less ambiguous: what's a
TestRecorder? Something that records tests?
We can't count on every child process reading everything we try to write to
it. And if the child terminates with WritePipe data still pending, unless we
explicitly suppress it, Posix will hit us with SIGPIPE. That would terminate
the calling process, boom. "Ignoring" it means APR gets the correct errno,
passes it back to us, we log it, etc.
Previously one might get process-terminated notification but still have to
wait for the child process's final data to arrive on one or more ReadPipes.
That required complex consumer timing logic to handle incomplete pending
ReadPipe data, e.g. a partial last line with no terminating newline. New code
guarantees that by the time LLProcess sends process-terminated notification,
all pending pipe data will have been buffered in ReadPipes.
Document LLProcess::ReadPipe::getPump() notification event; add "eof" key.
Add LLProcess::ReadPipe::getline() and read() convenience methods.
Add static LLProcess::getline() and basename() convenience methods, publishing
logic already present elsewhere.
Use ReadPipe::getline() and read() in unit tests.
Add unit test for "eof" event on ReadPipe::getPump().
Add unit test verifying that final data have been buffered by termination
notification event.
We want to verify the sequence:
LLInstanceTracker constructor adds instance to underlying container
Subclass constructor throws exception
LLInstanceTracker destructor removes instance from underlying container.
For the T* specialization (no string, or whatever, key), the original
getInstance() method simply returned the passed-in T* value. It was defined,
as the comments noted, for completeness of the analogy with the keyed
LLInstanceTracker specialization.
It turns out, though, that getInstance(T*) can still be useful to ask whether
the T* you have in hand still references a valid T instance. Support that
usage.
Lacking time to properly test new LLStringUtil::getTokens() against the
present (different!) command-line scanners in LLExternalEditor::tokenize() and
LLCommandLineParser::parseCommandLineString(), just annotate as future work
the goal of unifying them... SIGH.
This is an important differentiator between getTokens() and the present
LLCommandLineParser::parseCommandLineString() logic: you cannot currently
--set SomeVar to an empty string value because parseCommandLineString()
discards empty strings.
run_build_test.py already has the capability to set environment variables, and
we may as well direct it to set PYTHON to the running Python interpreter. That
completely eliminates one level of process wrapper.
We didn't have any tokenizer suitable for scanning something like a bash
command line. We do have a couple hacks, e.g. LLExternalEditor::tokenize() and
LLCommandLineParser::parseCommandLineString(). Both try to work around
boost::tokenizer limitations; but existing boost::tokenizer support just
doesn't address this case. Neither of the above is available as a general
scanner anyway, and parseCommandLineString() fails outright when passed "".
New getTokens() also distinguishes between "drop delimiters" (e.g. space,
return, newline) to be discarded from the token stream, versus "keep
delimiters" (e.g. "+-*/") to be returned as tokens in their own right.
There's an overload that honors escapes and a more efficient one that doesn't;
each has a convenience overload that returns the scanned string vector rather
than requiring a separate declaration.
Tweak and comment older getTokens() implementation.
Add unit tests for both old and new getTokens() implementations.
Break out StringVec and std::ostream << StringVec from
indra/llcommon/tests/listener.h to StringVec.h: that's coming in handy for a
number of different TUT test sources.
Clarify wording in some of the doc comments; be a bit more explicit about some
of the parameter fields.
Make some query methods 'const'.
Change default LLProcess::ReadPipe::getLimit() value to 0: don't post any
incoming data with notification event unless caller requests it. But do post
pertinent FILESLOT in case caller reuses same listener for both stdout and
stderr.
Use more idiomatic, readable syntax for accessing LLProcess::Params data.
If caller runs (e.g.) a Python script, it's not very helpful to a human log
reader to keep seeing LLProcess instances logged as /pathname/to/python (pid).
If caller is aware, the code can at least use the script name as the desc --
or maybe even a hint as to the script's purpose.
If caller doesn't explicitly pass a desc, at least shorten to just the
basename of the executable.
This way a caller need not spin on isRunning(); we can just listen for the
requested termination event.
Post a similar event containing error message if for any reason
LLProcess::create() failed to launch the child.
Add unit tests for both cases.
That is, trying to instantiate a ReadPipeImpl while another already existed
would throw an LLEventPump::DupPumpName exception. Fortunately this behavior
is easily bypassed.
If it's useful to have contains() to tell you whether incoming data contains a
particular substring, and if it's useful for contains() and peek() to accept
an offset within that data, then it's useful to allow you to get the offset of
a desired substring within that data. But of course a find() returning offset
needs something like std::string::npos for "not found"; borrow that
convention.
Support both find(const std::string&) and find(char); the latter permits a
more efficient implementation. In fact, make find(string) recognize a string
of length 1 and leverage the find(char) implementation.
Given that, reimplement contains(mumble) as shorthand for find(mumble) != npos.
Implement find() overloads using std::search() and std::find() on
boost::asio::streambuf character iterators, rather than copying to std::string
and then using string search like previous contains() implementation.
Reimplement WritePipeImpl::tick() and ReadPipeImpl::tick() to write/read
directly from/to boost::asio::streambuf data, instead of copying to/from a
temporary flat buffer.
As long as ReadPipeImpl::tick() keeps successfully filling buffers, keep
reading. Previous implementation would only handle a long child write over
successive tick() calls. Stop on read error or when we come up short.
Also add "len" key to event data on LLProcess::getPump(). If you've used
setLimit(), event["data"].length() may not reflect the length of the
accumulated data in the ReadPipe.
Add unit test with stdin/stdout handshake with child process.
In the course of re-enabling the indra/test tests last year, Log generalized a
workaround I'd introduced in llsdmessage_test.cpp. In Linux viewer land, a
test program trying to catch an expected exception can't seem to catch it by
its specific class (across the libllcommon.so boundary), but must instead
catch std::runtime_error and validate the typeid().name() string. Log added a
macro for this idiom in llevents_tut.cpp. Generalize that macro further for
normal-case processing as well, move it to a header file of its own and use it
in all known places -- plus the new exception-catching tests in
llprocess_test.cpp.
Add LLProcess::FileParam to specify how to construct each child's standard
file slot, with lots of comments about features designed but not yet
implemented. The point is to design it with enough flexibility to be able to
extend to foreseeable use cases.
Add LLProcess::Params::files to collect up to 3 FileParam items. Naturally
this extends the accepted LLSD syntax as well.
Implement type="" (child inherits parent file descriptor) and "pipe" (parent
constructs anonymous pipe to pass to child).
Add LLProcess::FILESLOT enum, plus methods:
getReadPipe(FILESLOT), getOptReadPipe(FILESLOT)
getWritePipe(), getOptWritePipe()
getPipeName(FILESLOT): placeholder implementation for now
Add LLProcess::ReadPipe and WritePipe classes, as returned by get*Pipe().
WritePipe supports get_ostream() method for streaming to child stdin.
ReadPipe supports get_istream() method for reading from child stdout/stderr.
It also provides getPump() returning LLEventPump& so interested parties can
listen for arrival of new data on the aforementioned std::istream.
For "pipe" slots, instantiate appropriate *Pipe class.
ReadPipe and WritePipe classes are pure virtual bases for ReadPipeImpl and
WritePipeImpl, respectively: all implementation data are hidden in the latter
classes, visible only in llprocess.cpp. In fact each *PipeImpl class registers
itself for "mainloop" ticks, attempting nonblocking I/O to the underlying
apr_file_t on each tick. Data are buffered in a boost::asio::streambuf, which
bridges between std::[io]stream and the APR I/O calls.
Sanity-test ReadPipeImpl by using a pipe to absorb the Python "SyntaxError"
output from the successful syntax_error test, rather than alarming the user.
Add first few unit tests for validating FileParam. More tests coming!
Nat Goodspeed