This changeset makes it possible to build the Second Life viewer using
Python 3. It is designed to be used with an equivalent Autobuild branch
so that a developer can compile without needing Python 2 on their
machine.
Breaking change: Python 2 support ending
Rather than supporting two versions of Python, including one that was
discontinued at the beginning of the year, this branch focuses on
pouring future effort into Python 3 only. As a result, scripts do not
need to be backwards compatible. This means that build environments,
be they on personal computers and on build agents, need to have a
compatible interpreter.
Notes
- SLVersionChecker will still use Python 2 on macOS
- Fixed the message template url used by template_verifier.py
Apparently, in previous Boost versions, boost::noncopyable was sneaking into
the namespace via other headers. Now the compiler complains about its absence
without an explicit #include.
For reasons not yet diagnosed, specifically in Mac Release builds, the tests
in test_httprequest.hpp consistently crash with a backtrace suggesting that
the worker thread is calling LLCore::HttpLibcurl::completeRequest() after the
foreground thread calls HttpRequest::destroyService().
Weirdly, even executing a tut::skip() call in every test<n>() function up to
the point of the crash does not eliminate the crash.
NickyD discovered that the substitute default allocator used for llcorehttp
tests was returning badly-aligned storage, which caused access violations on
alignment-sensitive data such as std::atomic. Thanks Nicky!!
Moreover, the llcorehttp test assertions regarding memory usage, well-
intentioned though they are, have been causing us trouble for years. Many have
already been disabled.
The problem is that use of test_allocator.h affected *everything* defined with
that header file's declarations visible. That inevitably included specific
functions in other subsystems. Those functions then (unintentionally) consumed
the special allocator, throwing off the memory tracking and making certain
memory-related assertions consistently fail.
This is a particular, observable bad effect of One Definition Rule violations.
Within a given program, C++ allows multiple definitions for the same entity,
but requires that all such definitions be the same. Partial visibility of the
global operator new() and operator delete() overrides meant that some
definitions of certain entities used the default global allocator, some used
llcorehttp's. There may have been other, more subtle bad effects of these ODR
violations.
If one wanted to reimplement verification of the memory consumption of
llcorehttp classes:
* Each llcorehttp class (for which memory tracking was desired) should declare
class-specific operator new() and operator delete() methods. Naturally,
these would all consume a central llcorehttp-specific allocator, but that
allocator should *not* be named global operator new().
* Presumably that would require runtime indirection to allow using the default
allocator in production while substituting the special allocator for tests.
* Recording and verifying the memory consumption in each test should be
performed in the test-object constructor and destructor, rather than being
sprinkled throughout the test<n>() methods.
* With that mechanism in place, the test object should provide methods to
adjust (or entirely disable) memory verification for a particular test.
* The test object should also provide a "yes, we're still consuming llcorehttp
memory" method to be used for spot checks in the middle of tests -- instead
of sprinkling in explicit comparisons as before.
* In fact, the llcorehttp test object in each test_*.hpp file should be
derived from a central llcorehttp test-object base class providing those
methods.
llcorehttp's test_allocator.{h,cpp} overrides global operator new(), operator
new[](), operator delete() and operator delete[](). The two operator new()
functions used to be declared with throw(std::bad_alloc). Worse, for VS 2013
and previous, we needed _THROW0() and _THROW1(std::bad_alloc) instead,
requiring #if logic.
But with dynamic throw declarations deprecated, we must actually remove those.
That obviates the THROW_BAD_ALLOC() / THROW_NOTHING() workarounds in
test_allocator.cpp.
In three different places we use the same pattern: an ssl_thread_id_callback()
function (a static member of LLCrashLogger, in that case) that used to be
passed to CRYPTO_set_id_callback() and therefore returned an unsigned long
representing the ID of the current thread.
But GetCurrentThread() is a HANDLE, an alias for a pointer, and you can't
uniquely cram a 64-bit pointer into an unsigned long.
Fortunately OpenSSL has a more modern API for retrieving thread ID. Pass
each ssl_thread_id_callback() function to CRYPTO_THREADID_set_callback()
instead, converting it to accept CRYPTO_THREADID* and call
CRYPTO_THREADID_set_pointer() or CRYPTO_THREADID_set_numeric() as appropriate().
LLThread::currentID() used to return a U32, a distinct unsigned value
incremented by explicitly constructing LLThread or by calling LLThread::
registerThreadID() early in a thread launched by other means. The latter
imposed an unobvious requirement on new code based on std::thread. Using
std:🧵:id instead delegates to the compiler/library the problem of
distinguishing threads launched by any means.
Change lots of explicit U32 declarations. Introduce LLThread::id_t typedef to
avoid having to run around fixing uses again if we later revisit this decision.
LLMutex, which stores an LLThread::id_t, wants a distinguished value meaning
NO_THREAD, and had an enum with that name. But as std:🧵:id promises
that the default-constructed value is distinct from every valid value,
NO_THREAD becomes unnecessary and goes away.
Because LLMutex now stores LLThread::id_t instead of U32, make llmutex.h
#include "llthread.h" instead of the other way around. This makes LLMutex an
incomplete type within llthread.h, so move LLThread::lockData() and
unlockData() to the .cpp file. Similarly, remove llrefcount.h's #include
"llmutex.h" to break circularity; instead forward-declare LLMutex.
It turns out that a number of source files assumed that #include "llthread.h"
would get the definition for LLMutex. Sprinkle #include "llmutex.h" as needed.
In the SAFE_SSL code in llcorehttp/httpcommon.cpp, there's an ssl_thread_id()
callback that returns an unsigned long to the SSL library. When LLThread::
currentID() was U32, we could simply return that. But std:🧵:id is very
deliberately opaque, and can't be reinterpret_cast to unsigned long.
Fortunately it can be hashed because std::hash is specialized with that type.
The crash can appear on some non-windows platforms (any LP64 model platforms).
Depending on alignment this can overwrite one word of the pointer `op` declared
above. Subsequently it will crash when later writing to memory through that
pointer
Andrey Lihatskiy