2020-01-16 20:35:17 +00:00
/*
* Copyright ( C ) 2006 - 2020 Apple Inc . All rights reserved .
* Copyright ( C ) 2007 - 2009 Torch Mobile , Inc .
* Copyright ( C ) 2010 , 2011 Research In Motion Limited . All rights reserved .
*
* Redistribution and use in source and binary forms , with or without
* modification , are permitted provided that the following conditions
* are met :
* 1. Redistributions of source code must retain the above copyright
* notice , this list of conditions and the following disclaimer .
* 2. Redistributions in binary form must reproduce the above copyright
* notice , this list of conditions and the following disclaimer in the
* documentation and / or other materials provided with the distribution .
*
* THIS SOFTWARE IS PROVIDED BY APPLE INC . ` ` AS IS ' ' AND ANY
* EXPRESS OR IMPLIED WARRANTIES , INCLUDING , BUT NOT LIMITED TO , THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
* PURPOSE ARE DISCLAIMED . IN NO EVENT SHALL APPLE INC . OR
* CONTRIBUTORS BE LIABLE FOR ANY DIRECT , INDIRECT , INCIDENTAL , SPECIAL ,
* EXEMPLARY , OR CONSEQUENTIAL DAMAGES ( INCLUDING , BUT NOT LIMITED TO ,
* PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES ; LOSS OF USE , DATA , OR
* PROFITS ; OR BUSINESS INTERRUPTION ) HOWEVER CAUSED AND ON ANY THEORY
* OF LIABILITY , WHETHER IN CONTRACT , STRICT LIABILITY , OR TORT
* ( INCLUDING NEGLIGENCE OR OTHERWISE ) ARISING IN ANY WAY OUT OF THE USE
* OF THIS SOFTWARE , EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE .
*/
# pragma once
2020-01-17 01:16:43 +00:00
# ifndef WTF_PLATFORM_GUARD_AGAINST_INDIRECT_INCLUSION
2020-01-16 20:35:17 +00:00
# error "Please #include <wtf / Platform.h> instead of this file directly."
# endif
/* USE() - use a particular third-party library or optional OS service */
# define USE(WTF_FEATURE) (defined USE_##WTF_FEATURE && USE_##WTF_FEATURE)
# if PLATFORM(COCOA)
# if defined __has_include && __has_include(<CoreFoundation / CFPriv.h>)
# define USE_APPLE_INTERNAL_SDK 1
# endif
# endif
/* Export macro support. Detects the attributes available for shared library symbol export
decorations . */
# if OS(WINDOWS) || (COMPILER_HAS_CLANG_DECLSPEC(dllimport) && COMPILER_HAS_CLANG_DECLSPEC(dllexport))
# define USE_DECLSPEC_ATTRIBUTE 1
# elif defined(__GNUC__)
# define USE_VISIBILITY_ATTRIBUTE 1
# endif
# if PLATFORM(COCOA)
# define USE_CG 1
# endif
2020-09-11 20:27:38 +00:00
# if PLATFORM(COCOA)
# define USE_CORE_TEXT 1
# endif
2020-01-16 20:35:17 +00:00
# if PLATFORM(COCOA)
# define USE_CA 1
# endif
Adding Experimental Feature Flags for CoreImage backed SVG/CSS Filters
https://bugs.webkit.org/show_bug.cgi?id=213578
Patch by Guowei Yang <guowei_yang@apple.com> on 2020-06-29
.:
Reviewed by Darin Adler, Simon Fraser, Myles C. Maxfield.
Preparing to implement CoreImage backed filter rendering
Needs Compiler guards and experimental feature guard.
* Source/cmake/WebKitFeatures.cmake: added definition of
CoreImage-accelerated filter rendering feature flag,
ENABLE_CORE_IMAGE_ACCELERATED_FILTER_RENDER
Source/WebCore:
Reviewed by Darin Adler, Simon Fraser, Myles C. Maxfield.
Preparing to implement CoreImage backed filter rendering
Needs Compiler guards and experimental feature guard.
No tests are required because this is just a feature flag set up
* page/Settings.yaml: added default settings for the feature flag.
Default value of the feature switch is off
Source/WebKit:
Reviewed by Darin Adler, Simon Fraser, Myles C. Maxfield.
Preparing to implement CoreImage backed filter rendering
Needs Compiler guards and experimental feature guard.
* Shared/WebPreferences.yaml: added feature flag definition
Source/WTF:
Reviewed by Darin Adler, Simon Fraser, Myles C. Maxfield.
Preparing to implement CoreImage backed filter rendering
Needs Compiler guards and experimental feature guard.
* wtf/PlatformEnableCocoa.h: Added definition of a feature flag,
ENABLE_CORE_IMAGE_ACCELERATED_FILTER_RENDER
* wtf/PlatformUse.h: Added compiler guard #define USE_CORE_IMAGE 1
to indicate whether CoreImage code is visible to the compiler
Canonical link: https://commits.webkit.org/226566@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@263718 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-06-30 01:59:29 +00:00
# if PLATFORM(COCOA)
# define USE_CORE_IMAGE 1
# endif
2020-01-16 20:35:17 +00:00
# if PLATFORM(GTK) || PLATFORM(WPE)
# define USE_GLIB 1
# endif
# if PLATFORM(GTK) || PLATFORM(WPE)
# define USE_FREETYPE 1
# endif
# if PLATFORM(GTK) || PLATFORM(WPE)
# define USE_HARFBUZZ 1
# endif
# if PLATFORM(GTK) || PLATFORM(WPE)
# define USE_SOUP 1
# endif
2021-01-05 20:02:58 +00:00
# if PLATFORM(GTK) || PLATFORM(WPE) || PLATFORM(HAIKU)
2020-01-16 20:35:17 +00:00
# define USE_WEBP 1
# endif
2021-01-05 20:02:58 +00:00
# if PLATFORM(HAIKU)
# define USE_HAIKU 1
# endif
2020-01-16 20:35:17 +00:00
/* On Windows, use QueryPerformanceCounter by default */
# if OS(WINDOWS)
# define USE_QUERY_PERFORMANCE_COUNTER 1
# endif
# if PLATFORM(COCOA)
# define USE_CF 1
# endif
# if PLATFORM(COCOA) || (PLATFORM(GTK) || PLATFORM(WPE))
# define USE_FILE_LOCK 1
# endif
# if PLATFORM(COCOA)
# define USE_FOUNDATION 1
# endif
# if PLATFORM(MAC)
# define USE_APPKIT 1
# endif
# if PLATFORM(MAC)
# define USE_PASSKIT 1
# endif
# if PLATFORM(MAC)
# define USE_PLUGIN_HOST_PROCESS 1
# endif
# if PLATFORM(IOS_FAMILY)
# define USE_UIKIT_EDITING 1
# endif
# if PLATFORM(IOS_FAMILY)
# define USE_WEB_THREAD 1
# endif
# if !defined(USE_UIKIT_KEYBOARD_ADDITIONS) && (PLATFORM(IOS) || PLATFORM(MACCATALYST))
# define USE_UIKIT_KEYBOARD_ADDITIONS 1
# endif
# if OS(UNIX)
# define USE_PTHREADS 1
# endif
# if OS(DARWIN) && !PLATFORM(GTK)
# define USE_ACCELERATE 1
# endif
2021-08-08 09:49:37 +00:00
# if OS(WINDOWS) || OS(HAIKU)
2020-01-16 20:35:17 +00:00
# define USE_SYSTEM_MALLOC 1
# endif
Allow Bitmap to use up to a UCPURegister word size for internal bit storage.
https://bugs.webkit.org/show_bug.cgi?id=211328
<rdar://problem/62755865>
Reviewed by Yusuke Suzuki.
Source/JavaScriptCore:
* assembler/CPU.h:
Source/WTF:
1. Moved the definition of CPURegister and UCPURegister down into WTF.
Added CPU(REGISTER64) and CPU(REGISTER32) for determining what size a CPU
general purpose register is.
2. Updated Bitmap so that it will automatically choose the minimal required
word size for the number of bits it needs to store. This means the Bitmap
can automatically choose a WordType from uint8_t up to UCPURegister.
Previously, the WordType is always uint32_t by default.
This should improve perf with use of Bitmap on 64-bit platforms. The size
optimization is necessary to prevent bloat on 64-bit platforms which would have
resulted if we simply set the default to always be UCPURegister.
3. Added a check in findRunOfZeros() for handling the edge case where the
requested runLength exceeds the bitmapSize.
4. Fixed a bug in count() that was unnecessarily casting the bits to unsigned
instead of just using the Bitmap WordType. As a result, when using a WordType
of uint64_t, it was discarding bits from the count.
5. Fixed invert() to leave the bits beyond bitmapSize untouched.
Fixed isFull() to ignore the bits beyond bitmapSize.
By fixing invert() to leave those bits as 0, isEmpty() and hash() will
continue to work. Otherwise, inverting those bits will cause isEmpty() to
always fail, and hash()'s result may be different for the same set of bit
values within bitmapSize.
isFull(), on the other hand, checks for set bits in the words. Since there
may be 0 valued bits beyond bitmapSize, isFull() needs to be fixed to ignore
those.
* WTF.xcodeproj/project.pbxproj:
* wtf/Bitmap.h:
(WTF::WordType>::invert):
(WTF::WordType>::findRunOfZeros const):
(WTF::WordType>::count const):
(WTF::WordType>::isFull const):
* wtf/CMakeLists.txt:
* wtf/PlatformCPU.h:
* wtf/PlatformUse.h:
* wtf/StdIntExtras.h: Copied from Source/WTF/wtf/StdIntExtras.h.
Tools:
Added API tests for WTF::Bitmap to make sure that Bitmap is behaving correctly.
Since Bitmap is used in critical infrastructure like the GC, it is important to
ensure that there are no latent bugs.
* TestWebKitAPI/CMakeLists.txt:
* TestWebKitAPI/TestWebKitAPI.xcodeproj/project.pbxproj:
* TestWebKitAPI/Tests/WTF/Bitmap.cpp: Added.
(TestWebKitAPI::countBits):
(TestWebKitAPI::testBitmapSize):
(TestWebKitAPI::testBitmapConstructedEmpty):
(TestWebKitAPI::testBitmapSetGet):
(TestWebKitAPI::testBitmapTestAndSet):
(TestWebKitAPI::testBitmapTestAndClear):
(TestWebKitAPI::testBitmapConcurrentTestAndSet):
(TestWebKitAPI::testBitmapConcurrentTestAndClear):
(TestWebKitAPI::testBitmapClear):
(TestWebKitAPI::testBitmapClearAll):
(TestWebKitAPI::testBitmapInvert):
(TestWebKitAPI::testBitmapFindRunOfZeros):
(TestWebKitAPI::testBitmapCount):
(TestWebKitAPI::testBitmapIsEmpty):
(TestWebKitAPI::testBitmapIsFull):
(TestWebKitAPI::testBitmapMerge):
(TestWebKitAPI::testBitmapFilter):
(TestWebKitAPI::testBitmapExclude):
(TestWebKitAPI::testBitmapConcurrentFilter):
(TestWebKitAPI::testBitmapSubsumes):
(TestWebKitAPI::testBitmapForEachSetBit):
(TestWebKitAPI::testBitmapFindBit):
(TestWebKitAPI::testBitmapIteration):
(TestWebKitAPI::testBitmapMergeAndClear):
(TestWebKitAPI::testBitmapSetAndClear):
(TestWebKitAPI::testBitmapOperatorEqual):
(TestWebKitAPI::testBitmapOperatorNotEqual):
(TestWebKitAPI::testBitmapHash):
(TestWebKitAPI::TEST):
LayoutTests:
editing/undo-manager/undo-manager-delete-stale-undo-items.html exposed a bug in
this patch. However, when a failure occurs, this test runs perpetually until it
times out. There's no need to do this. After a finite number of GC cycles,
unreachable objects should be collected. This is especially so because
GCController.collect() does a synchronous full GC.
Added a cap of 10 GC tries, and fail out if the test does not see the expected
result. This allows the test to fail fast and avoid the costly time out.
* editing/undo-manager/undo-manager-delete-stale-undo-items.html:
Canonical link: https://commits.webkit.org/224352@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@261179 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-05-05 17:08:29 +00:00
# if CPU(REGISTER64)
2020-01-16 20:35:17 +00:00
# define USE_JSVALUE64 1
# else
# define USE_JSVALUE32_64 1
# endif
Support an inlined representation in JSValue of small BigInts ("BigInt32")
https://bugs.webkit.org/show_bug.cgi?id=206182
Reviewed by Yusuke Suzuki.
JSTests:
I improved several of the tests to give more informative error messages in the process of fixing them.
More interestingly I had to modify "missing-exception-check-in-string-compare" because it relied on "s1 == s1" resolving ropes, and we now just return true.
* stress/big-int-division.js:
(testDiv):
* stress/big-int-left-shift-wrapped-value.js:
(assert.sameValue):
* stress/big-int-logical-not.js:
(assert):
* stress/big-int-mod-jit.js:
* stress/big-int-right-shift-general.js:
(testRightShift):
* stress/big-int-type-of-proven-type.js:
(assert):
* stress/compare-strict-eq-on-various-types.js:
(testAllTypesCall):
* stress/ftl-string-strict-equality.js:
* stress/missing-exception-check-in-string-compare.js:
Source/JavaScriptCore:
This patch attempts to optimize the performance of BigInts, when they are small (32 bit or less).
It works by inlining them into JSValue on 64-bit platforms, avoiding the allocation of a JSBigInt.
The bit pattern we use is 0000:XXXX:XXXX:0012
This representation works because of the following things:
- It cannot be confused with a Double or Integer thanks to the top bits
- It cannot be confused with a pointer to a Cell, thanks to bit 1 which is set to true
- It cannot be confused with a pointer to wasm thanks to bit 0 which is set to false
- It cannot be confused with true/false because bit 2 is set to false
- It cannot be confused for null/undefined because bit 4 is set to true
This entire change is gated by USE(BIGINT32), to make it easier to disable if it turns out to have bugs.
It should also make it much easier to verify if a given bug comes from it or from something else.
Note that in this patch we create BigInt32s when parsing small BigInt constants, and most operations (e.g. Add or BitOr) produce a BigInt32 if both of their operands are BigInt32,
but we don't produce a BigInt32 from for example the substraction/division of two large heap-allocated JSBigInts, even if the result fits in 32-bits.
As a result, small BigInts can now either be heap-allocated or inlined in the JSValue.
This patch includes a significant refactor of various slow paths, which are now grouped together in Operations.h
Because this increased the size of Operations.h significantly, I split the parts of Operations.h which are only used by the GC into Scribble.h, to avoid bloating compile times.
In the DFG and FTL we now have 3 UseKinds for BigInts: HeapBigIntUse, BigInt32Use and AnyBigIntUse.
The latter is useful when we know that we are receiving BigInts, but speculation indicates a mix of heap-allocated and small (inlined) big-ints.
Unfortunately, a naive implementation of this patch significantly regresses the performance of StrictEq (and its variants), as it is no longer true that a cell and a non-cell cannot be equal.
Before this patch, the code was jumping to a slow path if either:
- at least one operand is a double
- or both operands are cells
Now, it also needs to jump to the slow path if at least one is a cell.
To recover this performance cost, I significantly rewrote this code, from
if (left is Cell && right is Cell) {
if (left == right)
return true;
goto slowPath;
}
if (! left is Int32) {
if (left is Number)
goto slowPath
}
if (! right is Int32) {
if (right is Number)
goto slowPath
}
return left == right
To the following:
if (left is Double || right is Double)
goto slowPath
if (left == right)
return true;
if (left is Cell || right is Cell)
goto slowPath
return false;
I believe this to be faster than just replacing (left is Cell && right is Cell) by an ||, because I found a bit-trick to check (left is Double || right is Double) which should help reduce the pressure on the branch predictor.
Early JetStream2 tests appear to confirm that this patch is roughly neutral while it was a 0.5% regression before I used this trick, but the numbers are still too noisy, I plan to do more measurements before landing this patch.
I don't yet have performance numbers for this patch on a BigInt benchmark, I will get such numbers before trying to land it, but I'd like some review in the meantime.
* JavaScriptCore.xcodeproj/project.pbxproj:
* assembler/X86Assembler.h:
(JSC::X86Assembler::X86InstructionFormatter::SingleInstructionBufferWriter::memoryModRM):
* bytecode/ArithProfile.cpp:
(JSC::ArithProfile<BitfieldType>::emitObserveResult):
(JSC::ArithProfile<BitfieldType>::shouldEmitSetBigInt32 const):
(JSC::ArithProfile<BitfieldType>::shouldEmitSetHeapBigInt const):
(JSC::ArithProfile<BitfieldType>::emitSetHeapBigInt const):
(JSC::ArithProfile<BitfieldType>::emitSetBigInt32 const):
(WTF::printInternal):
* bytecode/ArithProfile.h:
(JSC::ObservedResults::didObserveNonInt32):
(JSC::ObservedResults::didObserveBigInt):
(JSC::ObservedResults::didObserveHeapBigInt):
(JSC::ObservedResults::didObserveBigInt32):
(JSC::ArithProfile::didObserveHeapBigInt const):
(JSC::ArithProfile::didObserveBigInt32 const):
(JSC::ArithProfile::setObservedHeapBigInt):
(JSC::ArithProfile::setObservedBigInt32):
(JSC::ArithProfile::observeResult):
* bytecode/BytecodeList.rb:
* bytecode/BytecodeLivenessAnalysisInlines.h:
* bytecode/BytecodeUseDef.cpp:
(JSC::computeUsesForBytecodeIndexImpl):
(JSC::computeDefsForBytecodeIndexImpl):
* bytecode/CodeBlock.cpp:
* bytecode/DataFormat.h:
* bytecode/MethodOfGettingAValueProfile.cpp:
(JSC::MethodOfGettingAValueProfile::emitReportValue const):
* bytecode/MethodOfGettingAValueProfile.h:
* bytecode/SpeculatedType.cpp:
(JSC::dumpSpeculation):
(JSC::speculationFromClassInfo):
(JSC::speculationFromStructure):
(JSC::speculationFromValue):
(JSC::speculationFromJSType):
(JSC::leastUpperBoundOfStrictlyEquivalentSpeculations):
* bytecode/SpeculatedType.h:
(JSC::isBigInt32Speculation):
(JSC::isHeapBigIntSpeculation):
(JSC::isBigIntSpeculation):
* bytecompiler/BytecodeGenerator.cpp:
(JSC::BytecodeGenerator::emitEqualityOpImpl):
(JSC::BytecodeGenerator::addBigIntConstant):
* bytecompiler/BytecodeGenerator.h:
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::isToThisAnIdentity):
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGByteCodeParser.cpp:
(JSC::DFG::ByteCodeParser::parseBlock):
* dfg/DFGCapabilities.cpp:
(JSC::DFG::capabilityLevel):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGConstantFoldingPhase.cpp:
(JSC::DFG::ConstantFoldingPhase::foldConstants):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
(JSC::DFG::FixupPhase::fixupToThis):
(JSC::DFG::FixupPhase::fixupToNumeric):
(JSC::DFG::FixupPhase::observeUseKindOnNode):
(JSC::DFG::FixupPhase::fixupCompareStrictEqAndSameValue):
* dfg/DFGMayExit.cpp:
* dfg/DFGNode.h:
(JSC::DFG::Node::shouldSpeculateBigInt32):
(JSC::DFG::Node::shouldSpeculateHeapBigInt):
* dfg/DFGNodeType.h:
* dfg/DFGOSRExit.cpp:
(JSC::DFG::OSRExit::compileExit):
* dfg/DFGOSRExit.h:
* dfg/DFGOperations.cpp:
* dfg/DFGOperations.h:
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::SafeToExecuteEdge::operator()):
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compilePeepHoleBranch):
(JSC::DFG::SpeculativeJIT::compileValueBitNot):
(JSC::DFG::SpeculativeJIT::emitUntypedOrAnyBigIntBitOp):
(JSC::DFG::SpeculativeJIT::compileValueBitwiseOp):
(JSC::DFG::SpeculativeJIT::emitUntypedOrBigIntRightShiftBitOp):
(JSC::DFG::SpeculativeJIT::compileValueLShiftOp):
(JSC::DFG::SpeculativeJIT::compileValueBitRShift):
(JSC::DFG::SpeculativeJIT::compileShiftOp):
(JSC::DFG::SpeculativeJIT::compileValueAdd):
(JSC::DFG::SpeculativeJIT::compileValueSub):
(JSC::DFG::SpeculativeJIT::compileIncOrDec):
(JSC::DFG::SpeculativeJIT::compileValueNegate):
(JSC::DFG::SpeculativeJIT::compileValueMul):
(JSC::DFG::SpeculativeJIT::compileValueDiv):
(JSC::DFG::SpeculativeJIT::compileValueMod):
(JSC::DFG::SpeculativeJIT::compileValuePow):
(JSC::DFG::SpeculativeJIT::compare):
(JSC::DFG::SpeculativeJIT::compileStrictEq):
(JSC::DFG::SpeculativeJIT::speculateHeapBigInt):
(JSC::DFG::SpeculativeJIT::speculate):
(JSC::DFG::SpeculativeJIT::compileToNumeric):
(JSC::DFG::SpeculativeJIT::compileHeapBigIntEquality):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculateBigInt32Operand::SpeculateBigInt32Operand):
(JSC::DFG::SpeculateBigInt32Operand::~SpeculateBigInt32Operand):
(JSC::DFG::SpeculateBigInt32Operand::edge const):
(JSC::DFG::SpeculateBigInt32Operand::node const):
(JSC::DFG::SpeculateBigInt32Operand::gpr):
(JSC::DFG::SpeculateBigInt32Operand::use):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::fillJSValue):
(JSC::DFG::SpeculativeJIT::nonSpeculativePeepholeStrictEq):
(JSC::DFG::SpeculativeJIT::nonSpeculativeNonPeepholeStrictEq):
(JSC::DFG::SpeculativeJIT::fillSpeculateInt32Internal):
(JSC::DFG::SpeculativeJIT::fillSpeculateCell):
(JSC::DFG::SpeculativeJIT::fillSpeculateBoolean):
(JSC::DFG::SpeculativeJIT::speculateBigInt32):
(JSC::DFG::SpeculativeJIT::speculateAnyBigInt):
(JSC::DFG::SpeculativeJIT::fillSpeculateBigInt32):
(JSC::DFG::SpeculativeJIT::compileBigInt32Compare):
(JSC::DFG::SpeculativeJIT::compilePeepHoleBigInt32Branch):
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* dfg/DFGUseKind.cpp:
(WTF::printInternal):
* dfg/DFGUseKind.h:
(JSC::DFG::typeFilterFor):
(JSC::DFG::isCell):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLCommonValues.cpp:
(JSC::FTL::CommonValues::initializeConstants):
* ftl/FTLCommonValues.h:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileValueAdd):
(JSC::FTL::DFG::LowerDFGToB3::compileValueSub):
(JSC::FTL::DFG::LowerDFGToB3::compileValueMul):
(JSC::FTL::DFG::LowerDFGToB3::compileBinaryMathIC):
(JSC::FTL::DFG::LowerDFGToB3::compileValueDiv):
(JSC::FTL::DFG::LowerDFGToB3::compileValueMod):
(JSC::FTL::DFG::LowerDFGToB3::compileValuePow):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitNot):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitAnd):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitOr):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitXor):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitRShift):
(JSC::FTL::DFG::LowerDFGToB3::compileArithBitRShift):
(JSC::FTL::DFG::LowerDFGToB3::compileArithBitLShift):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitLShift):
(JSC::FTL::DFG::LowerDFGToB3::compileBitURShift):
(JSC::FTL::DFG::LowerDFGToB3::compileToNumeric):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareEq):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):
(JSC::FTL::DFG::LowerDFGToB3::compileIsBigInt):
(JSC::FTL::DFG::LowerDFGToB3::emitBinarySnippet):
(JSC::FTL::DFG::LowerDFGToB3::emitBinaryBitOpSnippet):
(JSC::FTL::DFG::LowerDFGToB3::boolify):
(JSC::FTL::DFG::LowerDFGToB3::buildTypeOf):
(JSC::FTL::DFG::LowerDFGToB3::lowHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::lowBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isNotBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::unboxBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::boxBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isNotAnyBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculate):
(JSC::FTL::DFG::LowerDFGToB3::isNotHeapBigIntUnknownWhetherCell):
(JSC::FTL::DFG::LowerDFGToB3::isNotHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::isHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculateHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculateHeapBigIntUnknownWhetherCell):
(JSC::FTL::DFG::LowerDFGToB3::speculateBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::speculateAnyBigInt):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStub):
* heap/HeapSnapshotBuilder.cpp:
(JSC::HeapSnapshotBuilder::json):
* heap/MarkedBlockInlines.h:
* heap/PreciseAllocation.cpp:
* inspector/agents/InspectorHeapAgent.cpp:
(Inspector::InspectorHeapAgent::getPreview):
* interpreter/Interpreter.cpp:
(JSC::sizeOfVarargs):
* jit/AssemblyHelpers.cpp:
(JSC::AssemblyHelpers::emitConvertValueToBoolean):
(JSC::AssemblyHelpers::branchIfValue):
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::branchIfBigInt32):
(JSC::AssemblyHelpers::branchIfBigInt32KnownNotNumber):
(JSC::AssemblyHelpers::branchIfNotBigInt32KnownNotNumber):
(JSC::AssemblyHelpers::branchIfHeapBigInt):
(JSC::AssemblyHelpers::branchIfNotHeapBigInt):
(JSC::AssemblyHelpers::unboxBigInt32):
(JSC::AssemblyHelpers::boxBigInt32):
(JSC::AssemblyHelpers::emitTypeOf):
* jit/JIT.cpp:
(JSC::JIT::privateCompileMainPass):
* jit/JIT.h:
* jit/JITArithmetic.cpp:
(JSC::JIT::emit_op_negate):
(JSC::JIT::emitSlow_op_negate):
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_is_big_int):
(JSC::JIT::compileOpStrictEq):
(JSC::JIT::compileOpStrictEqJump):
(JSC::JIT::emit_op_to_numeric):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_is_big_int):
(JSC::JIT::emit_op_to_numeric):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* llint/LLIntOfflineAsmConfig.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ParserArena.cpp:
(JSC::IdentifierArena::makeBigIntDecimalIdentifier):
* runtime/ArrayPrototype.cpp:
* runtime/BigIntConstructor.cpp:
(JSC::toBigInt):
(JSC::callBigIntConstructor):
* runtime/BigIntObject.cpp:
(JSC::BigIntObject::create):
(JSC::BigIntObject::finishCreation):
* runtime/BigIntObject.h:
* runtime/BigIntPrototype.cpp:
(JSC::toThisBigIntValue):
(JSC::bigIntProtoFuncToStringImpl):
* runtime/CommonSlowPaths.cpp:
(JSC::SLOW_PATH_DECL):
(JSC::updateArithProfileForUnaryArithOp):
(JSC::updateArithProfileForBinaryArithOp):
* runtime/JSBigInt.cpp:
(JSC::JSBigInt::createStructure):
(JSC::JSBigInt::parseInt):
(JSC::JSBigInt::stringToBigInt):
(JSC::JSBigInt::inc):
(JSC::JSBigInt::dec):
(JSC::JSBigInt::bitwiseAnd):
(JSC::JSBigInt::toStringGeneric):
(JSC::JSBigInt::equalsToNumber):
(JSC::JSBigInt::equalsToInt32):
* runtime/JSBigInt.h:
(JSC::asHeapBigInt):
* runtime/JSCJSValue.cpp:
(JSC::JSValue::toNumberSlowCase const):
(JSC::JSValue::toObjectSlowCase const):
(JSC::JSValue::toThisSlowCase const):
(JSC::JSValue::synthesizePrototype const):
(JSC::JSValue::dumpInContextAssumingStructure const):
(JSC::JSValue::dumpForBacktrace const):
(JSC::JSValue::toStringSlowCase const):
* runtime/JSCJSValue.h:
* runtime/JSCJSValueInlines.h:
(JSC::JSValue::JSValue):
(JSC::JSValue::asHeapBigInt const):
(JSC::JSValue::isBigInt const):
(JSC::JSValue::isHeapBigInt const):
(JSC::JSValue::isBigInt32 const):
(JSC::JSValue::bigInt32AsInt32 const):
(JSC::JSValue::isPrimitive const):
(JSC::JSValue::getPrimitiveNumber):
(JSC::JSValue::toNumeric const):
(JSC::JSValue::toBigIntOrInt32 const):
(JSC::JSValue::equalSlowCaseInline):
(JSC::JSValue::strictEqualForCells):
(JSC::JSValue::strictEqual):
(JSC::JSValue::pureStrictEqual):
(JSC::JSValue::pureToBoolean const):
* runtime/JSCell.cpp:
(JSC::JSCell::put):
(JSC::JSCell::putByIndex):
(JSC::JSCell::toPrimitive const):
(JSC::JSCell::getPrimitiveNumber const):
(JSC::JSCell::toNumber const):
(JSC::JSCell::toObjectSlow const):
* runtime/JSCell.h:
* runtime/JSCellInlines.h:
(JSC::JSCell::isHeapBigInt const):
(JSC::JSCell::toBoolean const):
(JSC::JSCell::pureToBoolean const):
* runtime/JSString.h:
(JSC::JSValue::toBoolean const):
* runtime/JSType.cpp:
(WTF::printInternal):
* runtime/JSType.h:
* runtime/JSTypeInfo.h:
* runtime/ObjectInitializationScope.cpp:
* runtime/Operations.cpp:
(JSC::jsAddSlowCase):
(JSC::jsIsObjectTypeOrNull):
* runtime/Operations.h:
(JSC::compareBigIntToOtherPrimitive):
(JSC::bigIntCompare):
(JSC::jsLess):
(JSC::jsLessEq):
(JSC::arithmeticBinaryOp):
(JSC::jsSub):
(JSC::jsMul):
(JSC::jsDiv):
(JSC::jsRemainder):
(JSC::jsPow):
(JSC::jsInc):
(JSC::jsDec):
(JSC::jsBitwiseNot):
(JSC::shift):
(JSC::jsLShift):
(JSC::jsRShift):
(JSC::bitwiseBinaryOp):
(JSC::jsBitwiseAnd):
(JSC::jsBitwiseOr):
(JSC::jsBitwiseXor):
* runtime/Scribble.h: Copied from Source/JavaScriptCore/runtime/BigIntObject.h.
(JSC::scribbleFreeCells):
(JSC::isScribbledValue):
(JSC::scribble):
* runtime/StructureInlines.h:
(JSC::prototypeForLookupPrimitiveImpl):
Source/WTF:
Add a USE(BIGINT32) flag.
* wtf/PlatformUse.h:
Canonical link: https://commits.webkit.org/223591@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@260331 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-04-19 02:20:59 +00:00
# if USE(JSVALUE64)
2020-07-27 20:30:03 +00:00
/* FIXME: Enable BIGINT32 optimization again after we ensure Speedometer2 and JetStream2 regressions are fixed. */
/* https://bugs.webkit.org/show_bug.cgi?id=214777 */
# define USE_BIGINT32 0
Support an inlined representation in JSValue of small BigInts ("BigInt32")
https://bugs.webkit.org/show_bug.cgi?id=206182
Reviewed by Yusuke Suzuki.
JSTests:
I improved several of the tests to give more informative error messages in the process of fixing them.
More interestingly I had to modify "missing-exception-check-in-string-compare" because it relied on "s1 == s1" resolving ropes, and we now just return true.
* stress/big-int-division.js:
(testDiv):
* stress/big-int-left-shift-wrapped-value.js:
(assert.sameValue):
* stress/big-int-logical-not.js:
(assert):
* stress/big-int-mod-jit.js:
* stress/big-int-right-shift-general.js:
(testRightShift):
* stress/big-int-type-of-proven-type.js:
(assert):
* stress/compare-strict-eq-on-various-types.js:
(testAllTypesCall):
* stress/ftl-string-strict-equality.js:
* stress/missing-exception-check-in-string-compare.js:
Source/JavaScriptCore:
This patch attempts to optimize the performance of BigInts, when they are small (32 bit or less).
It works by inlining them into JSValue on 64-bit platforms, avoiding the allocation of a JSBigInt.
The bit pattern we use is 0000:XXXX:XXXX:0012
This representation works because of the following things:
- It cannot be confused with a Double or Integer thanks to the top bits
- It cannot be confused with a pointer to a Cell, thanks to bit 1 which is set to true
- It cannot be confused with a pointer to wasm thanks to bit 0 which is set to false
- It cannot be confused with true/false because bit 2 is set to false
- It cannot be confused for null/undefined because bit 4 is set to true
This entire change is gated by USE(BIGINT32), to make it easier to disable if it turns out to have bugs.
It should also make it much easier to verify if a given bug comes from it or from something else.
Note that in this patch we create BigInt32s when parsing small BigInt constants, and most operations (e.g. Add or BitOr) produce a BigInt32 if both of their operands are BigInt32,
but we don't produce a BigInt32 from for example the substraction/division of two large heap-allocated JSBigInts, even if the result fits in 32-bits.
As a result, small BigInts can now either be heap-allocated or inlined in the JSValue.
This patch includes a significant refactor of various slow paths, which are now grouped together in Operations.h
Because this increased the size of Operations.h significantly, I split the parts of Operations.h which are only used by the GC into Scribble.h, to avoid bloating compile times.
In the DFG and FTL we now have 3 UseKinds for BigInts: HeapBigIntUse, BigInt32Use and AnyBigIntUse.
The latter is useful when we know that we are receiving BigInts, but speculation indicates a mix of heap-allocated and small (inlined) big-ints.
Unfortunately, a naive implementation of this patch significantly regresses the performance of StrictEq (and its variants), as it is no longer true that a cell and a non-cell cannot be equal.
Before this patch, the code was jumping to a slow path if either:
- at least one operand is a double
- or both operands are cells
Now, it also needs to jump to the slow path if at least one is a cell.
To recover this performance cost, I significantly rewrote this code, from
if (left is Cell && right is Cell) {
if (left == right)
return true;
goto slowPath;
}
if (! left is Int32) {
if (left is Number)
goto slowPath
}
if (! right is Int32) {
if (right is Number)
goto slowPath
}
return left == right
To the following:
if (left is Double || right is Double)
goto slowPath
if (left == right)
return true;
if (left is Cell || right is Cell)
goto slowPath
return false;
I believe this to be faster than just replacing (left is Cell && right is Cell) by an ||, because I found a bit-trick to check (left is Double || right is Double) which should help reduce the pressure on the branch predictor.
Early JetStream2 tests appear to confirm that this patch is roughly neutral while it was a 0.5% regression before I used this trick, but the numbers are still too noisy, I plan to do more measurements before landing this patch.
I don't yet have performance numbers for this patch on a BigInt benchmark, I will get such numbers before trying to land it, but I'd like some review in the meantime.
* JavaScriptCore.xcodeproj/project.pbxproj:
* assembler/X86Assembler.h:
(JSC::X86Assembler::X86InstructionFormatter::SingleInstructionBufferWriter::memoryModRM):
* bytecode/ArithProfile.cpp:
(JSC::ArithProfile<BitfieldType>::emitObserveResult):
(JSC::ArithProfile<BitfieldType>::shouldEmitSetBigInt32 const):
(JSC::ArithProfile<BitfieldType>::shouldEmitSetHeapBigInt const):
(JSC::ArithProfile<BitfieldType>::emitSetHeapBigInt const):
(JSC::ArithProfile<BitfieldType>::emitSetBigInt32 const):
(WTF::printInternal):
* bytecode/ArithProfile.h:
(JSC::ObservedResults::didObserveNonInt32):
(JSC::ObservedResults::didObserveBigInt):
(JSC::ObservedResults::didObserveHeapBigInt):
(JSC::ObservedResults::didObserveBigInt32):
(JSC::ArithProfile::didObserveHeapBigInt const):
(JSC::ArithProfile::didObserveBigInt32 const):
(JSC::ArithProfile::setObservedHeapBigInt):
(JSC::ArithProfile::setObservedBigInt32):
(JSC::ArithProfile::observeResult):
* bytecode/BytecodeList.rb:
* bytecode/BytecodeLivenessAnalysisInlines.h:
* bytecode/BytecodeUseDef.cpp:
(JSC::computeUsesForBytecodeIndexImpl):
(JSC::computeDefsForBytecodeIndexImpl):
* bytecode/CodeBlock.cpp:
* bytecode/DataFormat.h:
* bytecode/MethodOfGettingAValueProfile.cpp:
(JSC::MethodOfGettingAValueProfile::emitReportValue const):
* bytecode/MethodOfGettingAValueProfile.h:
* bytecode/SpeculatedType.cpp:
(JSC::dumpSpeculation):
(JSC::speculationFromClassInfo):
(JSC::speculationFromStructure):
(JSC::speculationFromValue):
(JSC::speculationFromJSType):
(JSC::leastUpperBoundOfStrictlyEquivalentSpeculations):
* bytecode/SpeculatedType.h:
(JSC::isBigInt32Speculation):
(JSC::isHeapBigIntSpeculation):
(JSC::isBigIntSpeculation):
* bytecompiler/BytecodeGenerator.cpp:
(JSC::BytecodeGenerator::emitEqualityOpImpl):
(JSC::BytecodeGenerator::addBigIntConstant):
* bytecompiler/BytecodeGenerator.h:
* dfg/DFGAbstractInterpreterInlines.h:
(JSC::DFG::isToThisAnIdentity):
(JSC::DFG::AbstractInterpreter<AbstractStateType>::executeEffects):
* dfg/DFGByteCodeParser.cpp:
(JSC::DFG::ByteCodeParser::parseBlock):
* dfg/DFGCapabilities.cpp:
(JSC::DFG::capabilityLevel):
* dfg/DFGClobberize.h:
(JSC::DFG::clobberize):
* dfg/DFGConstantFoldingPhase.cpp:
(JSC::DFG::ConstantFoldingPhase::foldConstants):
* dfg/DFGDoesGC.cpp:
(JSC::DFG::doesGC):
* dfg/DFGFixupPhase.cpp:
(JSC::DFG::FixupPhase::fixupNode):
(JSC::DFG::FixupPhase::fixupToThis):
(JSC::DFG::FixupPhase::fixupToNumeric):
(JSC::DFG::FixupPhase::observeUseKindOnNode):
(JSC::DFG::FixupPhase::fixupCompareStrictEqAndSameValue):
* dfg/DFGMayExit.cpp:
* dfg/DFGNode.h:
(JSC::DFG::Node::shouldSpeculateBigInt32):
(JSC::DFG::Node::shouldSpeculateHeapBigInt):
* dfg/DFGNodeType.h:
* dfg/DFGOSRExit.cpp:
(JSC::DFG::OSRExit::compileExit):
* dfg/DFGOSRExit.h:
* dfg/DFGOperations.cpp:
* dfg/DFGOperations.h:
* dfg/DFGPredictionPropagationPhase.cpp:
* dfg/DFGSafeToExecute.h:
(JSC::DFG::SafeToExecuteEdge::operator()):
(JSC::DFG::safeToExecute):
* dfg/DFGSpeculativeJIT.cpp:
(JSC::DFG::SpeculativeJIT::compilePeepHoleBranch):
(JSC::DFG::SpeculativeJIT::compileValueBitNot):
(JSC::DFG::SpeculativeJIT::emitUntypedOrAnyBigIntBitOp):
(JSC::DFG::SpeculativeJIT::compileValueBitwiseOp):
(JSC::DFG::SpeculativeJIT::emitUntypedOrBigIntRightShiftBitOp):
(JSC::DFG::SpeculativeJIT::compileValueLShiftOp):
(JSC::DFG::SpeculativeJIT::compileValueBitRShift):
(JSC::DFG::SpeculativeJIT::compileShiftOp):
(JSC::DFG::SpeculativeJIT::compileValueAdd):
(JSC::DFG::SpeculativeJIT::compileValueSub):
(JSC::DFG::SpeculativeJIT::compileIncOrDec):
(JSC::DFG::SpeculativeJIT::compileValueNegate):
(JSC::DFG::SpeculativeJIT::compileValueMul):
(JSC::DFG::SpeculativeJIT::compileValueDiv):
(JSC::DFG::SpeculativeJIT::compileValueMod):
(JSC::DFG::SpeculativeJIT::compileValuePow):
(JSC::DFG::SpeculativeJIT::compare):
(JSC::DFG::SpeculativeJIT::compileStrictEq):
(JSC::DFG::SpeculativeJIT::speculateHeapBigInt):
(JSC::DFG::SpeculativeJIT::speculate):
(JSC::DFG::SpeculativeJIT::compileToNumeric):
(JSC::DFG::SpeculativeJIT::compileHeapBigIntEquality):
* dfg/DFGSpeculativeJIT.h:
(JSC::DFG::SpeculateBigInt32Operand::SpeculateBigInt32Operand):
(JSC::DFG::SpeculateBigInt32Operand::~SpeculateBigInt32Operand):
(JSC::DFG::SpeculateBigInt32Operand::edge const):
(JSC::DFG::SpeculateBigInt32Operand::node const):
(JSC::DFG::SpeculateBigInt32Operand::gpr):
(JSC::DFG::SpeculateBigInt32Operand::use):
* dfg/DFGSpeculativeJIT32_64.cpp:
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGSpeculativeJIT64.cpp:
(JSC::DFG::SpeculativeJIT::fillJSValue):
(JSC::DFG::SpeculativeJIT::nonSpeculativePeepholeStrictEq):
(JSC::DFG::SpeculativeJIT::nonSpeculativeNonPeepholeStrictEq):
(JSC::DFG::SpeculativeJIT::fillSpeculateInt32Internal):
(JSC::DFG::SpeculativeJIT::fillSpeculateCell):
(JSC::DFG::SpeculativeJIT::fillSpeculateBoolean):
(JSC::DFG::SpeculativeJIT::speculateBigInt32):
(JSC::DFG::SpeculativeJIT::speculateAnyBigInt):
(JSC::DFG::SpeculativeJIT::fillSpeculateBigInt32):
(JSC::DFG::SpeculativeJIT::compileBigInt32Compare):
(JSC::DFG::SpeculativeJIT::compilePeepHoleBigInt32Branch):
(JSC::DFG::SpeculativeJIT::compile):
* dfg/DFGStrengthReductionPhase.cpp:
(JSC::DFG::StrengthReductionPhase::handleNode):
* dfg/DFGUseKind.cpp:
(WTF::printInternal):
* dfg/DFGUseKind.h:
(JSC::DFG::typeFilterFor):
(JSC::DFG::isCell):
* ftl/FTLCapabilities.cpp:
(JSC::FTL::canCompile):
* ftl/FTLCommonValues.cpp:
(JSC::FTL::CommonValues::initializeConstants):
* ftl/FTLCommonValues.h:
* ftl/FTLLowerDFGToB3.cpp:
(JSC::FTL::DFG::LowerDFGToB3::compileNode):
(JSC::FTL::DFG::LowerDFGToB3::compileValueAdd):
(JSC::FTL::DFG::LowerDFGToB3::compileValueSub):
(JSC::FTL::DFG::LowerDFGToB3::compileValueMul):
(JSC::FTL::DFG::LowerDFGToB3::compileBinaryMathIC):
(JSC::FTL::DFG::LowerDFGToB3::compileValueDiv):
(JSC::FTL::DFG::LowerDFGToB3::compileValueMod):
(JSC::FTL::DFG::LowerDFGToB3::compileValuePow):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitNot):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitAnd):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitOr):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitXor):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitRShift):
(JSC::FTL::DFG::LowerDFGToB3::compileArithBitRShift):
(JSC::FTL::DFG::LowerDFGToB3::compileArithBitLShift):
(JSC::FTL::DFG::LowerDFGToB3::compileValueBitLShift):
(JSC::FTL::DFG::LowerDFGToB3::compileBitURShift):
(JSC::FTL::DFG::LowerDFGToB3::compileToNumeric):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareEq):
(JSC::FTL::DFG::LowerDFGToB3::compileCompareStrictEq):
(JSC::FTL::DFG::LowerDFGToB3::compileIsBigInt):
(JSC::FTL::DFG::LowerDFGToB3::emitBinarySnippet):
(JSC::FTL::DFG::LowerDFGToB3::emitBinaryBitOpSnippet):
(JSC::FTL::DFG::LowerDFGToB3::boolify):
(JSC::FTL::DFG::LowerDFGToB3::buildTypeOf):
(JSC::FTL::DFG::LowerDFGToB3::lowHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::lowBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isNotBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::unboxBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::boxBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::isNotAnyBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculate):
(JSC::FTL::DFG::LowerDFGToB3::isNotHeapBigIntUnknownWhetherCell):
(JSC::FTL::DFG::LowerDFGToB3::isNotHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::isHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculateHeapBigInt):
(JSC::FTL::DFG::LowerDFGToB3::speculateHeapBigIntUnknownWhetherCell):
(JSC::FTL::DFG::LowerDFGToB3::speculateBigInt32):
(JSC::FTL::DFG::LowerDFGToB3::speculateAnyBigInt):
* ftl/FTLOSRExitCompiler.cpp:
(JSC::FTL::compileStub):
* heap/HeapSnapshotBuilder.cpp:
(JSC::HeapSnapshotBuilder::json):
* heap/MarkedBlockInlines.h:
* heap/PreciseAllocation.cpp:
* inspector/agents/InspectorHeapAgent.cpp:
(Inspector::InspectorHeapAgent::getPreview):
* interpreter/Interpreter.cpp:
(JSC::sizeOfVarargs):
* jit/AssemblyHelpers.cpp:
(JSC::AssemblyHelpers::emitConvertValueToBoolean):
(JSC::AssemblyHelpers::branchIfValue):
* jit/AssemblyHelpers.h:
(JSC::AssemblyHelpers::branchIfBigInt32):
(JSC::AssemblyHelpers::branchIfBigInt32KnownNotNumber):
(JSC::AssemblyHelpers::branchIfNotBigInt32KnownNotNumber):
(JSC::AssemblyHelpers::branchIfHeapBigInt):
(JSC::AssemblyHelpers::branchIfNotHeapBigInt):
(JSC::AssemblyHelpers::unboxBigInt32):
(JSC::AssemblyHelpers::boxBigInt32):
(JSC::AssemblyHelpers::emitTypeOf):
* jit/JIT.cpp:
(JSC::JIT::privateCompileMainPass):
* jit/JIT.h:
* jit/JITArithmetic.cpp:
(JSC::JIT::emit_op_negate):
(JSC::JIT::emitSlow_op_negate):
* jit/JITOpcodes.cpp:
(JSC::JIT::emit_op_is_big_int):
(JSC::JIT::compileOpStrictEq):
(JSC::JIT::compileOpStrictEqJump):
(JSC::JIT::emit_op_to_numeric):
* jit/JITOpcodes32_64.cpp:
(JSC::JIT::emit_op_is_big_int):
(JSC::JIT::emit_op_to_numeric):
* jit/JITOperations.cpp:
* jit/JITOperations.h:
* llint/LLIntOfflineAsmConfig.h:
* llint/LowLevelInterpreter.asm:
* llint/LowLevelInterpreter64.asm:
* parser/ParserArena.cpp:
(JSC::IdentifierArena::makeBigIntDecimalIdentifier):
* runtime/ArrayPrototype.cpp:
* runtime/BigIntConstructor.cpp:
(JSC::toBigInt):
(JSC::callBigIntConstructor):
* runtime/BigIntObject.cpp:
(JSC::BigIntObject::create):
(JSC::BigIntObject::finishCreation):
* runtime/BigIntObject.h:
* runtime/BigIntPrototype.cpp:
(JSC::toThisBigIntValue):
(JSC::bigIntProtoFuncToStringImpl):
* runtime/CommonSlowPaths.cpp:
(JSC::SLOW_PATH_DECL):
(JSC::updateArithProfileForUnaryArithOp):
(JSC::updateArithProfileForBinaryArithOp):
* runtime/JSBigInt.cpp:
(JSC::JSBigInt::createStructure):
(JSC::JSBigInt::parseInt):
(JSC::JSBigInt::stringToBigInt):
(JSC::JSBigInt::inc):
(JSC::JSBigInt::dec):
(JSC::JSBigInt::bitwiseAnd):
(JSC::JSBigInt::toStringGeneric):
(JSC::JSBigInt::equalsToNumber):
(JSC::JSBigInt::equalsToInt32):
* runtime/JSBigInt.h:
(JSC::asHeapBigInt):
* runtime/JSCJSValue.cpp:
(JSC::JSValue::toNumberSlowCase const):
(JSC::JSValue::toObjectSlowCase const):
(JSC::JSValue::toThisSlowCase const):
(JSC::JSValue::synthesizePrototype const):
(JSC::JSValue::dumpInContextAssumingStructure const):
(JSC::JSValue::dumpForBacktrace const):
(JSC::JSValue::toStringSlowCase const):
* runtime/JSCJSValue.h:
* runtime/JSCJSValueInlines.h:
(JSC::JSValue::JSValue):
(JSC::JSValue::asHeapBigInt const):
(JSC::JSValue::isBigInt const):
(JSC::JSValue::isHeapBigInt const):
(JSC::JSValue::isBigInt32 const):
(JSC::JSValue::bigInt32AsInt32 const):
(JSC::JSValue::isPrimitive const):
(JSC::JSValue::getPrimitiveNumber):
(JSC::JSValue::toNumeric const):
(JSC::JSValue::toBigIntOrInt32 const):
(JSC::JSValue::equalSlowCaseInline):
(JSC::JSValue::strictEqualForCells):
(JSC::JSValue::strictEqual):
(JSC::JSValue::pureStrictEqual):
(JSC::JSValue::pureToBoolean const):
* runtime/JSCell.cpp:
(JSC::JSCell::put):
(JSC::JSCell::putByIndex):
(JSC::JSCell::toPrimitive const):
(JSC::JSCell::getPrimitiveNumber const):
(JSC::JSCell::toNumber const):
(JSC::JSCell::toObjectSlow const):
* runtime/JSCell.h:
* runtime/JSCellInlines.h:
(JSC::JSCell::isHeapBigInt const):
(JSC::JSCell::toBoolean const):
(JSC::JSCell::pureToBoolean const):
* runtime/JSString.h:
(JSC::JSValue::toBoolean const):
* runtime/JSType.cpp:
(WTF::printInternal):
* runtime/JSType.h:
* runtime/JSTypeInfo.h:
* runtime/ObjectInitializationScope.cpp:
* runtime/Operations.cpp:
(JSC::jsAddSlowCase):
(JSC::jsIsObjectTypeOrNull):
* runtime/Operations.h:
(JSC::compareBigIntToOtherPrimitive):
(JSC::bigIntCompare):
(JSC::jsLess):
(JSC::jsLessEq):
(JSC::arithmeticBinaryOp):
(JSC::jsSub):
(JSC::jsMul):
(JSC::jsDiv):
(JSC::jsRemainder):
(JSC::jsPow):
(JSC::jsInc):
(JSC::jsDec):
(JSC::jsBitwiseNot):
(JSC::shift):
(JSC::jsLShift):
(JSC::jsRShift):
(JSC::bitwiseBinaryOp):
(JSC::jsBitwiseAnd):
(JSC::jsBitwiseOr):
(JSC::jsBitwiseXor):
* runtime/Scribble.h: Copied from Source/JavaScriptCore/runtime/BigIntObject.h.
(JSC::scribbleFreeCells):
(JSC::isScribbledValue):
(JSC::scribble):
* runtime/StructureInlines.h:
(JSC::prototypeForLookupPrimitiveImpl):
Source/WTF:
Add a USE(BIGINT32) flag.
* wtf/PlatformUse.h:
Canonical link: https://commits.webkit.org/223591@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@260331 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-04-19 02:20:59 +00:00
# endif
2020-01-16 20:35:17 +00:00
/* FIXME: This name should be more specific if it is only for use with CallFrame* */
/* Use __builtin_frame_address(1) to get CallFrame* */
# if COMPILER(GCC_COMPATIBLE) && (CPU(ARM64) || CPU(X86_64))
# define USE_BUILTIN_FRAME_ADDRESS 1
# endif
2021-03-18 00:07:54 +00:00
# if OS(DARWIN) && CPU(ARM64) && HAVE(REMAP_JIT)
2020-04-08 09:38:51 +00:00
# define USE_EXECUTE_ONLY_JIT_WRITE_FUNCTION 1
# endif
2020-01-16 20:35:17 +00:00
# if PLATFORM(IOS)
# define USE_PASSKIT 1
# endif
# if PLATFORM(IOS)
# define USE_QUICK_LOOK 1
# endif
# if PLATFORM(IOS)
# define USE_SYSTEM_PREVIEW 1
# endif
# if PLATFORM(COCOA)
# define USE_AVFOUNDATION 1
# endif
# if PLATFORM(COCOA)
# define USE_PROTECTION_SPACE_AUTH_CALLBACK 1
# endif
# if PLATFORM(COCOA)
# define USE_METAL 1
# endif
2021-05-09 11:33:21 +00:00
# if PLATFORM(GTK) || PLATFORM(WPE)
2020-01-16 20:35:17 +00:00
# define USE_UNIX_DOMAIN_SOCKETS 1
# endif
# if !defined(USE_IMLANG_FONT_LINK2)
# define USE_IMLANG_FONT_LINK2 1
# endif
2020-02-17 21:55:51 +00:00
# if PLATFORM(MAC) && __MAC_OS_X_VERSION_MAX_ALLOWED < 101500
# define USE_AV_SAMPLE_BUFFER_DISPLAY_LAYER 1
# endif
2020-01-16 20:35:17 +00:00
# if PLATFORM(COCOA) || PLATFORM(GTK)
# define USE_UNIFIED_TEXT_CHECKING 1
# endif
# if PLATFORM(MAC)
# define USE_AUTOMATIC_TEXT_REPLACEMENT 1
# endif
# if PLATFORM(MAC)
/* Some platforms provide UI for suggesting autocorrection. */
# define USE_AUTOCORRECTION_PANEL 1
# endif
# if PLATFORM(COCOA)
/* Some platforms use spelling and autocorrection markers to provide visual cue. On such platform, if word with marker is edited, we need to remove the marker. */
# define USE_MARKER_REMOVAL_UPON_EDITING 1
# endif
# if PLATFORM(MAC)
# define USE_INSERTION_UNDO_GROUPING 1
# endif
# if PLATFORM(COCOA)
# define USE_AUDIO_SESSION 1
# endif
# if PLATFORM(COCOA)
# define USE_MEDIATOOLBOX 1
# endif
# if PLATFORM(COCOA)
# define USE_OS_LOG 1
# endif
2021-07-13 01:22:14 +00:00
# if PLATFORM(COCOA)
2020-01-16 20:35:17 +00:00
# define USE_OS_STATE 1
# endif
# if !defined(WTF_DEFAULT_EVENT_LOOP)
# define WTF_DEFAULT_EVENT_LOOP 1
# endif
# if WTF_DEFAULT_EVENT_LOOP
# if USE(GLIB)
/* Use GLib's event loop abstraction. Primarily GTK port uses it. */
# define USE_GLIB_EVENT_LOOP 1
# elif OS(WINDOWS)
/* Use Windows message pump abstraction.
* Even if the port is AppleWin , we use the Windows message pump system for the event loop ,
* so that USE ( WINDOWS_EVENT_LOOP ) & & USE ( CF ) can be true .
* And PLATFORM ( WIN ) and PLATFORM ( GTK ) are exclusive . If the port is GTK ,
* PLATFORM ( WIN ) should be false . And in that case , GLib ' s event loop is used .
*/
# define USE_WINDOWS_EVENT_LOOP 1
# elif PLATFORM(COCOA)
/* OS X and IOS. Use CoreFoundation & GCD abstraction. */
# define USE_COCOA_EVENT_LOOP 1
2021-01-05 20:02:58 +00:00
# elif PLATFORM(HAIKU)
# define USE_HAIKU_EVENT_LOOP 1
2020-01-16 20:35:17 +00:00
# else
# define USE_GENERIC_EVENT_LOOP 1
# endif
# endif
# if PLATFORM(COCOA)
# define USE_DICTATION_ALTERNATIVES 1
# endif
# if (PLATFORM(MAC) && USE(APPLE_INTERNAL_SDK) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 101500) || PLATFORM(IOS_FAMILY)
# define USE_SOURCE_APPLICATION_AUDIT_DATA 1
# endif
# if PLATFORM(COCOA) && USE(CA)
# define USE_IOSURFACE_CANVAS_BACKING_STORE 1
# endif
/* The override isn't needed on iOS family, as the default behavior is to not sniff. */
2020-05-05 05:51:47 +00:00
# if PLATFORM(MAC)
2020-01-16 20:35:17 +00:00
# define USE_CFNETWORK_CONTENT_ENCODING_SNIFFING_OVERRIDE 1
# endif
[GTK] Stop using gtk foreign drawing API to style form controls
https://bugs.webkit.org/show_bug.cgi?id=208129
Reviewed by Adrian Perez de Castro.
.:
Add a test case for datalist element.
* ManualTests/gtk/theme.html:
Source/WebCore:
It causes layout issues in some websites, it doesn't really work with all GTK themes and it won't be possible
with GTK4 because foreign drawing APIs have been removed. Instead, we can use the new custom style used by WPE
port, which is based on adwaita, but simplified to avoid the huge minimum control sizes, the usage of gradients
and transparencies, etc. We can still use the GTK API to get the selection colors, to keep some consistency with
the actual GTK theme, but that won't be possible with GTK4 either. This also means we won't be rendering
directly to the cairo context anymore (which was required by GTK foreign drawing), so we can use things like
threaded rendering in the GTK port now. This patch renames ScrollbarThemeWPE, RenderThemeWPE and ThemeWPE as
ScrollbarThemeAdwaita, RenderThemeAdwaita and ThemeAdwaita and adds ThemeGtk. GTK media controls CSS and
JavaScript files have been removed in favor of using the adwaita ones.
* Modules/mediacontrols/mediaControlsGtk.js: Removed.
* PlatformGTK.cmake:
* PlatformWPE.cmake:
* SourcesGTK.txt:
* SourcesWPE.txt:
* css/mediaControlsGtk.css: Removed.
* platform/adwaita/ScrollbarThemeAdwaita.cpp: Renamed from Source/WebCore/platform/wpe/ScrollbarThemeWPE.cpp.
(WebCore::ScrollbarThemeAdwaita::usesOverlayScrollbars const):
(WebCore::ScrollbarThemeAdwaita::scrollbarThickness):
(WebCore::ScrollbarThemeAdwaita::minimumThumbLength):
(WebCore::ScrollbarThemeAdwaita::hasButtons):
(WebCore::ScrollbarThemeAdwaita::hasThumb):
(WebCore::ScrollbarThemeAdwaita::backButtonRect):
(WebCore::ScrollbarThemeAdwaita::forwardButtonRect):
(WebCore::ScrollbarThemeAdwaita::trackRect):
(WebCore::ScrollbarThemeAdwaita::paint):
(WebCore::ScrollbarThemeAdwaita::handleMousePressEvent):
(WebCore::ScrollbarTheme::nativeTheme):
* platform/adwaita/ScrollbarThemeAdwaita.h: Copied from Source/WebCore/platform/wpe/ScrollbarThemeWPE.h.
* platform/adwaita/ThemeAdwaita.cpp: Renamed from Source/WebCore/platform/wpe/ThemeWPE.cpp.
(WebCore::Theme::singleton):
(WebCore::ThemeAdwaita::activeSelectionForegroundColor const):
(WebCore::ThemeAdwaita::activeSelectionBackgroundColor const):
(WebCore::ThemeAdwaita::inactiveSelectionForegroundColor const):
(WebCore::ThemeAdwaita::inactiveSelectionBackgroundColor const):
(WebCore::ThemeAdwaita::focusColor):
(WebCore::ThemeAdwaita::paintFocus):
(WebCore::ThemeAdwaita::paintArrow):
(WebCore::ThemeAdwaita::controlSize const):
(WebCore::ThemeAdwaita::paint):
(WebCore::ThemeAdwaita::paintCheckbox):
(WebCore::ThemeAdwaita::paintRadio):
(WebCore::ThemeAdwaita::paintButton):
(WebCore::ThemeAdwaita::paintSpinButton):
* platform/adwaita/ThemeAdwaita.h: Renamed from Source/WebCore/platform/wpe/ThemeWPE.h.
(WebCore::ThemeAdwaita::platformColorsDidChange):
* platform/graphics/cairo/CairoOperations.cpp:
(WebCore::Cairo::drawFocusRing):
* platform/graphics/cairo/GraphicsContextCairo.cpp:
(WebCore::GraphicsContext::drawFocusRing):
* platform/gtk/RenderThemeGadget.cpp: Removed.
* platform/gtk/RenderThemeGadget.h: Removed.
* platform/gtk/RenderThemeWidget.cpp: Removed.
* platform/gtk/RenderThemeWidget.h: Removed.
* platform/gtk/ScrollbarThemeGtk.cpp: Removed.
* platform/gtk/ScrollbarThemeGtk.h: Removed.
* platform/gtk/ThemeGtk.cpp: Added.
(WebCore::Theme::singleton):
(WebCore::ThemeGtk::ensurePlatformColors const):
(WebCore::ThemeGtk::platformColorsDidChange):
(WebCore::ThemeGtk::activeSelectionForegroundColor const):
(WebCore::ThemeGtk::activeSelectionBackgroundColor const):
(WebCore::ThemeGtk::inactiveSelectionForegroundColor const):
(WebCore::ThemeGtk::inactiveSelectionBackgroundColor const):
* platform/gtk/ThemeGtk.h: Renamed from Source/WebCore/platform/wpe/ScrollbarThemeWPE.h.
* rendering/RenderThemeAdwaita.cpp: Renamed from Source/WebCore/platform/wpe/RenderThemeWPE.cpp.
(WebCore::RenderTheme::singleton):
(WebCore::RenderThemeAdwaita::supportsFocusRing const):
(WebCore::RenderThemeAdwaita::shouldHaveCapsLockIndicator const):
(WebCore::RenderThemeAdwaita::platformActiveSelectionBackgroundColor const):
(WebCore::RenderThemeAdwaita::platformInactiveSelectionBackgroundColor const):
(WebCore::RenderThemeAdwaita::platformActiveSelectionForegroundColor const):
(WebCore::RenderThemeAdwaita::platformInactiveSelectionForegroundColor const):
(WebCore::RenderThemeAdwaita::platformActiveListBoxSelectionBackgroundColor const):
(WebCore::RenderThemeAdwaita::platformInactiveListBoxSelectionBackgroundColor const):
(WebCore::RenderThemeAdwaita::platformActiveListBoxSelectionForegroundColor const):
(WebCore::RenderThemeAdwaita::platformInactiveListBoxSelectionForegroundColor const):
(WebCore::RenderThemeAdwaita::platformFocusRingColor const):
(WebCore::RenderThemeAdwaita::platformColorsDidChange):
(WebCore::RenderThemeAdwaita::extraDefaultStyleSheet):
(WebCore::RenderThemeAdwaita::extraMediaControlsStyleSheet):
(WebCore::RenderThemeAdwaita::mediaControlsScript):
(WebCore::RenderThemeAdwaita::paintTextField):
(WebCore::RenderThemeAdwaita::paintTextArea):
(WebCore::RenderThemeAdwaita::paintSearchField):
(WebCore::RenderThemeAdwaita::popupInternalPaddingBox const):
(WebCore::RenderThemeAdwaita::paintMenuList):
(WebCore::RenderThemeAdwaita::paintMenuListButtonDecorations):
(WebCore::RenderThemeAdwaita::animationRepeatIntervalForProgressBar const):
(WebCore::RenderThemeAdwaita::animationDurationForProgressBar const):
(WebCore::RenderThemeAdwaita::progressBarRectForBounds const):
(WebCore::RenderThemeAdwaita::paintProgressBar):
(WebCore::RenderThemeAdwaita::paintSliderTrack):
(WebCore::RenderThemeAdwaita::adjustSliderThumbSize const):
(WebCore::RenderThemeAdwaita::paintSliderThumb):
(WebCore::RenderThemeAdwaita::paintMediaSliderTrack):
(WebCore::RenderThemeAdwaita::paintMediaVolumeSliderTrack):
(WebCore::RenderThemeAdwaita::sliderTickSize const):
(WebCore::RenderThemeAdwaita::sliderTickOffsetFromTrackCenter const):
(WebCore::RenderThemeAdwaita::adjustListButtonStyle const):
* rendering/RenderThemeAdwaita.h: Renamed from Source/WebCore/platform/wpe/RenderThemeWPE.h.
* rendering/RenderThemeGtk.cpp:
(WebCore::RenderThemeGtk::updateCachedSystemFontDescription const):
(WebCore::RenderThemeGtk::caretBlinkInterval const):
* rendering/RenderThemeGtk.h:
Source/WebKit:
* WebProcess/WebPage/gtk/WebPageGtk.cpp:
(WebKit::WebPage::themeDidChange): Notify RenderTheme about the theme change to clear the colors cache.
Source/WTF:
Enable USE_NEW_THEME for the GTK port.
* wtf/PlatformUse.h:
Canonical link: https://commits.webkit.org/221120@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@257299 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2020-02-25 09:55:52 +00:00
# if PLATFORM(MAC) || PLATFORM(WPE) || PLATFORM(GTK)
2020-01-16 20:35:17 +00:00
/* FIXME: This really needs a descriptive name, this "new theme" was added in 2008. */
# define USE_NEW_THEME 1
# endif
# if PLATFORM(IOS) || PLATFORM(MACCATALYST)
# define USE_UICONTEXTMENU 1
# endif
# if PLATFORM(IOS_FAMILY) || (!(defined(USE_SYSTEM_MALLOC) && USE_SYSTEM_MALLOC) && (OS(LINUX) && (PLATFORM(GTK) || PLATFORM(WPE))))
# define USE_BMALLOC_MEMORY_FOOTPRINT_API 1
# endif
2020-01-28 01:30:41 +00:00
2020-02-09 01:29:05 +00:00
# if !defined(USE_PLATFORM_REGISTERS_WITH_PROFILE) && OS(DARWIN) && CPU(ARM64) && defined(__LP64__)
# define USE_PLATFORM_REGISTERS_WITH_PROFILE 1
# endif
2020-01-28 01:30:41 +00:00
# if OS(DARWIN) && !USE(PLATFORM_REGISTERS_WITH_PROFILE) && CPU(ARM64)
# define USE_DARWIN_REGISTER_MACROS 1
# endif
2020-02-06 23:31:17 +00:00
2020-08-06 00:58:34 +00:00
# if PLATFORM(COCOA) && !(PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 110000)
2020-08-11 04:23:51 +00:00
# define USE_CTFONTSHAPEGLYPHS 1
2020-02-06 23:31:17 +00:00
# endif
2020-09-05 18:07:19 +00:00
# if PLATFORM(COCOA) && (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 110000)
# define USE_CTFONTGETADVANCES_WORKAROUND 1
# endif
2020-11-04 02:31:56 +00:00
2021-07-28 18:57:11 +00:00
# if (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 120000) \
| | ( PLATFORM ( IOS_FAMILY ) & & __IPHONE_OS_VERSION_MIN_REQUIRED < 150000 ) \
| | ( PLATFORM ( WATCHOS ) & & ! PLATFORM ( IOS_FAMILY_SIMULATOR ) & & __WATCH_OS_VERSION_MIN_REQUIRED < 80000 ) \
| | ( PLATFORM ( APPLETV ) & & ! PLATFORM ( IOS_FAMILY_SIMULATOR ) & & __TV_OS_VERSION_MIN_REQUIRED < 150000 )
2021-07-21 04:23:21 +00:00
# define USE_CTFONTSHAPEGLYPHS_WORKAROUND 1
# endif
2021-08-20 03:21:05 +00:00
# if (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 120000) \
| | ( PLATFORM ( IOS_FAMILY ) & & __IPHONE_OS_VERSION_MIN_REQUIRED < 150000 ) \
| | ( PLATFORM ( WATCHOS ) & & ! PLATFORM ( IOS_FAMILY_SIMULATOR ) & & __WATCH_OS_VERSION_MIN_REQUIRED < 80000 ) \
| | ( PLATFORM ( APPLETV ) & & ! PLATFORM ( IOS_FAMILY_SIMULATOR ) & & __TV_OS_VERSION_MIN_REQUIRED < 150000 )
# define USE_NON_VARIABLE_SYSTEM_FONT 1
# endif
2020-11-11 21:09:42 +00:00
# if PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED < 101500
# define USE_LEGACY_CFNETWORK_DOWNLOADS 1
# endif
2020-11-19 23:08:56 +00:00
# if (PLATFORM(MAC) && __MAC_OS_X_VERSION_MIN_REQUIRED >= 110000) \
| | ( PLATFORM ( MACCATALYST ) & & __IPHONE_OS_VERSION_MIN_REQUIRED > = 140000 )
# if USE(APPLE_INTERNAL_SDK)
/* Always use the macro on internal builds */
# define USE_PTHREAD_JIT_PERMISSIONS_API 0
# else
# define USE_PTHREAD_JIT_PERMISSIONS_API 1
# endif
# endif
2020-11-20 04:32:46 +00:00
# if PLATFORM(COCOA)
# define USE_OPENXR 0
2021-05-06 17:05:23 +00:00
# define USE_IOSURFACE_FOR_XR_LAYER_DATA 1
2021-05-14 00:35:23 +00:00
# if !defined(HAVE_WEBXR_INTERNALS) && !HAVE(WEBXR_INTERNALS)
# define USE_EMPTYXR 1
# endif
2020-11-20 04:32:46 +00:00
# endif
2021-01-14 15:31:19 +00:00
# if PLATFORM(IOS_FAMILY)
# define USE_SANDBOX_EXTENSIONS_FOR_CACHE_AND_TEMP_DIRECTORY_ACCESS 1
# endif
New malloc algorithm
https://bugs.webkit.org/show_bug.cgi?id=226404
Reviewed by Yusuke Suzuki.
Source/bmalloc:
This change replaces bmalloc with libpas, by importing libpas and making it expose API that
looks like the current bmalloc API. Libpas replaces everything bmalloc gave us: the main
bmalloc API, the Gigacage API, and the IsoHeap API. Libpas also replaces the JSC
ExecutableAllocator (more on that in the JSC ChangeLog). I've been working on libpas for about
three years, and you'll be able to see the age of some of it in the copyrights.
Libpas is a totally new malloc implementation that focuses on allowing for the creation of lots
of isolated heaps, and making it possible for those heaps to provide certain type guarantees.
I'm not going to summarize everything that it does in this changelog; I'll mostly focus on
things that are interesting to its use in WebKit.
Libpas tries hard to make having lots of heaps cheap both by having a good isolated heap
implementation and by offsetting the cost by having a bunch of space saving improvements for all
kinds of heaps. Libpas heaps can be configured to be either very compact or very fast. In
WebKit, we configure libpas to be as fast as possible, except for the jit_heap, where we
configure it to be as compact as possible. The fast configuration relies on the segregated heap
for most allocations, while the compact configuration relies on bitfit for most configurations.
It's possible to use both in combination with any size cut-off before you switch to bitfit.
Libpas's segregated heap is competitive to bmalloc in speed. This patch is a speed-up on AS, and
it's only enabled on AS for now. Libpas is still a regression relative to bmalloc on some
devices that I have measured, so it should probably stay enabled on AS only for now. This patch
ought to be a 1% speed-up on Speedometer and MotionMark and be neutral elsewhere.
When it comes to memory usage, libpas's segregated heap is a 19% improvement on membuster and a
11% improvement on RAMification on AS, with most other devices exhibiting similar wins if you
enable libpas on them.
Here's the basic design:
- Libpas makes extreme use of generic programming in C, sometimes using macros, but mostly using
a technique where ALWAYS_INLINE functions are passed structs with function pointers to other
ALWAYS_INLINE functions, sometimes many layers deep. I've perfected this so that I can cause
outlining to happen wherever I want without losing the specialization, and I can cause
anything I want to become generic (i.e. it gets a pointer to that struct with function
pointers and it really has to call them).
This allows libpas to have many copies of the same algorithm, but configured differently.
That's why, for example, I was able to make libpas's bitfit obey ExecutableAllocator rules
like that you cannot put any metadata inside the memory you're managing. That's also how
libpas "scales" it's algorithm: there are the small configurations, and then with a different
struct, we get the medium configurations (larger overall page size, ability to decommit each
physical page within the logical page), and the marge configurations (optimized for even
bigger object sizes).
- Allocation uses thread-local caches that hold on to up to a page of memory. This is similar
overall to bmalloc, but rather different in the details. When allocating from that memory (see
bmalloc_heap_inlines.h, then pas_try_allocate_intrinsic_primitive.h, then
pas_try_allocate_common.h, then pas_local_allocator_inlines.h), libpas uses either bump
pointer or bit search. The bit search is optimized to death. Libpas does lots of clever things
with bits: for example, each bit represents just a "minalign" worth of memory (so like 16
bytes), and bits are set at the beginnings of objects. Lots of interesting bit manipulation
happens on the slow path to quickly turn this into a bitvector where the bits are set at
free object beginnings, which is what the allocator consumes (see for example
pas_local_allocator_scan_bits_to_set_up_free_bits() in pas_local_allocator_inlines.h). This
saves libpas from having to use multiplications on the allocation and deallocation paths.
Allocation fast paths are super fast and require no locks. Slow paths use fine-grained
locking so the likelihood of contention is very low. Each page has a lock, and they do crazy
things with their locks: pages will all share the same lock so long as that's efficient, but
will pick up different locks if that causes contention, and then they will share the same
lock again if contention goes away.
Allocation caches can be reclaimed by the scavenger. This is done using an
ask-nicely-then-more-firmly kind of mechanism: first the scavenger will just ask allocators
that haven't been used in a while to return their memory, and the thread will service this
request on the next allocation slow path for any thread-local allocator. But if the thread
doesn't do this, then the scavenger will eventually suspend the thread and take its caches.
- Deallocation uses a log. The log flush amortizes lock acquisition and memory access to libpas
metadata. This is quite a bit like bmalloc. The differences are: libpas may acquire multiple
locks during deallocation log flush, but because pages try to share locks, that usually
doesn't happen. Usually, all pages that were most recently used by some CPU will share the
same lock. The other difference is that the libpas scavenger can flush deallocation logs, and
does so on every tick.
The design described so far is for the libpas segregated heap, which is the most
performance-optimized heap, as well as the heap most suited for isoheaps. Segregated heap has
many isoheap optimizations, like allowing multiple heaps to share different slabs of the same
page. But libpas also has two other heap kinds:
- Bitfit. The bitfit heap uses a bit-per-minalign but uses it to implement first-fit. Allocation
searches for a span of set "free" bits and then clears them. Deallocation uses a second
bitvector, the "object ends" bits, to find the end of the object and then sets the whole range
of free bits. This allocator uses per-page locking and a clever short-circuiting algorithm
to find the first page that will have a contiguous slab of free memory big enough for the
size you're trying to allocate. This is much slower than the segregated heap, but also much
faster than the other option, which is the libpas large allocator. Bitfit is the most
space-efficient heap in libpas, except for super large objects. The space efficiency comes
from the fact that it's first-fit. It should be noted that segregated is also first-fit, but
within size classes, so segregated gets more external fragmentation than bitfit.
This patch causes us to use bitfit for the mini mode. We also use bitfit for marge allocations
(so bigger than segregated medium but too small for large) and for jit_heap.
I think the most interesting file for bitfit is pas_bitfit_page_inlines.h.
- Large. This is a first-fit allocator implemented using cartesian trees. Something special
about it is that it can do type-safe array allocations with complex combinations of memalign.
For example, you can tell it to create a heap that allocates arrays of a type whose size is
42, and then it will guarantee you that no matter how big or how small of an array you
allocate, no element of that array will ever be an "offset overlap" with a previously freed
element. So, it won't create any type confusions. It will guarantee this even if you did
memalign with any alignment (I needed to use extended GCD for this -- see pas_extended_gcd.c
and pas_coalign.c). The large algorithm is implemented generically
(pas_generic_large_free_heap.h), so it can use either cartesian trees
(pas_fast_large_free_heap.c) or just an array (pas_simple_large_free_heap.c). The large heap
is used internally in libpas a lot, for example for the bootstrap heap, which is how all of
libpas gets its memory. One particularly neat thing about this is that the bootstrap heap
allocates a freelist array inside of the memory it is managing using a kind of wild
metacircular design.
One of the main space saving features is that libpas has a fine-grained-locked LRU decommit
policy using a 10Hz scavenger thread. This scavenger incurs almost no perf cost but is able
to return a ton of memory. At any tick, it will decommit any totally free pages that haven't
been used in the last 300ms. This includes pages used for internal libpas metadata, like the
thread-local caches. Decommitting pages holds locks that other threads are unlikely to want to
grab (because of policies that actively avoid it), so decommit happens with minimal interference
to the running program.
Libpas is written in C and strongly assumes that the C compiler is modern enough to fixpoint
inlining of always-inline functions with the right other optimizations. Llvm does this, which
enables a wonderful style of generic programming. The fact that the same code can be used
for generic, specialized-and-inline, and specialized-but-out-of-line functions is great. I use
WebKit C++ style (so foo* p rather than foo *p) but with lower_case_identifiers for anything
that isn't a macro. Anything that isn't static is prefixed pas_, unless it's some specific
heap configuration, like the bmalloc one, which prefixes every non-static with bmalloc_.
Libpas includes a giant test suite. Lots of the heap configurations (iso_test, thingy,
minalign32, pagesize64k) are just for testing. The test suite has both unit tests and chaos
tests as well as everything in between. The test suite is written in C++.
This glues libpas into the DebugHeap mechanism in bmalloc, so Malloc=1 will give you system
malloc. Libpas supports libmalloc enumeration, and this patch exposes it (though you need a
shared cache rebuild to enjoy the benefits).
All of bmalloc's external-facing API are replaced with libpas when the BUSE(LIBPAS) flag is
set (so the bmalloc::XYZ functions, Gigacage::XYZ, and IsoHeap<> and friends). This also
exposes the jit_heap API for the ExecutableAllocator to use.
Notes for porting: this uses lots of Darwin APIs, like the APIs for getting an approximate time
very quickly, APIs for fast TLS, thread suspension, and probably other stuff. It'll be hard,
but possible, to port to Linux. It may never perform as well on Linux as it does on Darwin.
It's also worth noting that libpas strongly assumes 64-bit, that the CPU can do 128-bit
compare-and-swap, that there is no big downside to misaligned loads and stores, and that the
memory model is either the one in ARM64 or the one on x86_64 (or something not weaker than
either of them). Trying to make libpas work on 32-bit CPUs will quickly run into cases where
some pointer math doesn't work right because it was written strongly assuming properties
unique to 64-bit CPUs (like that some number of top bits are not used for the address), as well
as cases where lock-free algorithms are using uintptr_t for a versioning scheme (which may not
be as reliable as you want on 32-bit).
* CMakeLists.txt:
* Configurations/Base.xcconfig:
* bmalloc.xcodeproj/project.pbxproj:
* bmalloc/BPlatform.h:
* bmalloc/DebugHeap.cpp:
(bmalloc::DebugHeap::tryGetSlow):
(pas_debug_heap_is_enabled):
(pas_debug_heap_malloc):
(pas_debug_heap_memalign):
(pas_debug_heap_realloc):
(pas_debug_heap_free):
* bmalloc/DebugHeap.h:
(bmalloc::debugHeapDisabled):
(bmalloc::DebugHeap::tryGet):
(bmalloc::DebugHeap::getExisting):
* bmalloc/Gigacage.cpp:
(Gigacage::ensureGigacage):
(Gigacage::allocBase):
(Gigacage::size):
* bmalloc/Gigacage.h:
* bmalloc/GigacageConfig.h:
(Gigacage::Config::allocBasePtr const):
(Gigacage::Config::setAllocBasePtr):
(Gigacage::Config::allocSize const):
(Gigacage::Config::setAllocSize):
* bmalloc/Heap.cpp:
(bmalloc::Heap::gigacageSize):
* bmalloc/Heap.h:
* bmalloc/IsoHeap.cpp: Added.
(bmalloc::api::isoAllocate):
(bmalloc::api::isoTryAllocate):
(bmalloc::api::isoDeallocate):
* bmalloc/IsoHeap.h:
(bmalloc::api::IsoHeap::IsoHeap):
(bmalloc::api::IsoHeap::allocate):
(bmalloc::api::IsoHeap::tryAllocate):
(bmalloc::api::IsoHeap::deallocate):
(bmalloc::api::IsoHeap::scavenge):
(bmalloc::api::IsoHeap::initialize):
(bmalloc::api::IsoHeap::isInitialized):
* bmalloc/IsoHeapImplInlines.h:
(bmalloc::IsoHeapImpl<Config>::IsoHeapImpl):
* bmalloc/IsoHeapInlines.h:
* bmalloc/IsoMallocFallback.cpp: Added.
(bmalloc::IsoMallocFallback::tryMalloc):
(bmalloc::IsoMallocFallback::tryFree):
* bmalloc/IsoMallocFallback.h: Copied from Source/bmalloc/bmalloc/DebugHeap.h.
(bmalloc::IsoMallocFallback::shouldTryToFallBack):
(bmalloc::IsoMallocFallback::MallocResult::MallocResult):
* bmalloc/IsoTLS.cpp:
(bmalloc::IsoTLS::determineMallocFallbackState): Deleted.
* bmalloc/IsoTLS.h:
* bmalloc/IsoTLSInlines.h:
(bmalloc::IsoTLS::allocateSlow):
(bmalloc::IsoTLS::deallocateSlow):
* bmalloc/PerThread.h:
(bmalloc::PerThreadStorage<PerHeapKind<Cache>>::init):
* bmalloc/bmalloc.cpp:
(bmalloc::api::tryLargeZeroedMemalignVirtual):
(bmalloc::api::freeLargeVirtual):
(bmalloc::api::scavengeThisThread):
(bmalloc::api::scavenge):
(bmalloc::api::setScavengerThreadQOSClass):
(bmalloc::api::commitAlignedPhysical):
(bmalloc::api::decommitAlignedPhysical):
(bmalloc::api::enableMiniMode):
(bmalloc::api::disableScavenger):
* bmalloc/bmalloc.h:
(bmalloc::api::heapForKind):
(bmalloc::api::tryMalloc):
(bmalloc::api::malloc):
(bmalloc::api::tryMemalign):
(bmalloc::api::memalign):
(bmalloc::api::tryRealloc):
(bmalloc::api::realloc):
(bmalloc::api::free):
(bmalloc::api::scavengeThisThread): Deleted.
* libpas/.gitignore: Added.
* libpas/build.sh: Added.
* libpas/build_and_test.sh: Added.
* libpas/common.sh: Added.
* libpas/libpas.xcodeproj/project.pbxproj: Added.
* libpas/scripts/tally_verifier_output: Added.
* libpas/src/chaos/Chaos.cpp: Added.
(std::Object::Object):
(std::Packet::~Packet):
(std::Locker::Locker):
(std::Locker::~Locker):
(std::threadMain):
(main):
* libpas/src/libpas/bmalloc_heap.c: Added.
(bmalloc_try_allocate):
(bmalloc_try_allocate_with_alignment):
(bmalloc_try_allocate_zeroed):
(bmalloc_allocate):
(bmalloc_allocate_with_alignment):
(bmalloc_allocate_zeroed):
(bmalloc_try_reallocate):
(bmalloc_reallocate):
(bmalloc_try_iso_allocate):
(bmalloc_iso_allocate):
(bmalloc_heap_ref_get_heap):
(bmalloc_try_allocate_auxiliary):
(bmalloc_allocate_auxiliary):
(bmalloc_try_allocate_auxiliary_zeroed):
(bmalloc_allocate_auxiliary_zeroed):
(bmalloc_try_allocate_auxiliary_with_alignment):
(bmalloc_allocate_auxiliary_with_alignment):
(bmalloc_try_reallocate_auxiliary):
(bmalloc_reallocate_auxiliary):
(bmalloc_deallocate):
(bmalloc_force_auxiliary_heap_into_reserved_memory):
* libpas/src/libpas/bmalloc_heap.h: Added.
* libpas/src/libpas/bmalloc_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(bmalloc_heap_config_activate):
* libpas/src/libpas/bmalloc_heap_config.h: Added.
* libpas/src/libpas/bmalloc_heap_inlines.h: Added.
(bmalloc_try_allocate_inline):
(bmalloc_try_allocate_with_alignment_inline):
(bmalloc_try_allocate_zeroed_inline):
(bmalloc_allocate_inline):
(bmalloc_allocate_with_alignment_inline):
(bmalloc_allocate_zeroed_inline):
(bmalloc_try_reallocate_inline):
(bmalloc_reallocate_inline):
(bmalloc_try_iso_allocate_inline):
(bmalloc_iso_allocate_inline):
(bmalloc_try_allocate_auxiliary_inline):
(bmalloc_allocate_auxiliary_inline):
(bmalloc_try_allocate_auxiliary_zeroed_inline):
(bmalloc_allocate_auxiliary_zeroed_inline):
(bmalloc_try_allocate_auxiliary_with_alignment_inline):
(bmalloc_allocate_auxiliary_with_alignment_inline):
(bmalloc_try_reallocate_auxiliary_inline):
(bmalloc_reallocate_auxiliary_inline):
(bmalloc_deallocate_inline):
* libpas/src/libpas/bmalloc_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/bmalloc_heap_ref.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/hotbit_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(hotbit_try_allocate):
(hotbit_try_allocate_with_alignment):
(hotbit_try_reallocate):
(hotbit_deallocate):
* libpas/src/libpas/hotbit_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/hotbit_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(hotbit_heap_config_activate):
* libpas/src/libpas/hotbit_heap_config.h: Added.
* libpas/src/libpas/hotbit_heap_inlines.h: Added.
(hotbit_try_allocate_inline):
(hotbit_try_allocate_with_alignment_inline):
(hotbit_try_reallocate_inline):
(hotbit_deallocate_inline):
* libpas/src/libpas/hotbit_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap.c: Added.
(iso_try_allocate_common_primitive):
(iso_try_allocate_common_primitive_with_alignment):
(iso_try_allocate_common_primitive_zeroed):
(iso_allocate_common_primitive):
(iso_allocate_common_primitive_with_alignment):
(iso_allocate_common_primitive_zeroed):
(iso_try_reallocate_common_primitive):
(iso_reallocate_common_primitive):
(iso_try_allocate_dynamic_primitive):
(iso_try_allocate_dynamic_primitive_with_alignment):
(iso_try_allocate_dynamic_primitive_zeroed):
(iso_try_reallocate_dynamic_primitive):
(iso_heap_ref_construct):
(iso_try_allocate):
(iso_allocate):
(iso_try_allocate_array):
(iso_allocate_array):
(iso_try_allocate_array_zeroed):
(iso_allocate_array_zeroed):
(iso_try_reallocate_array):
(iso_reallocate_array):
(iso_heap_ref_get_heap):
(iso_primitive_heap_ref_construct):
(iso_try_allocate_primitive):
(iso_allocate_primitive):
(iso_try_allocate_primitive_zeroed):
(iso_allocate_primitive_zeroed):
(iso_try_allocate_primitive_with_alignment):
(iso_allocate_primitive_with_alignment):
(iso_try_reallocate_primitive):
(iso_reallocate_primitive):
(iso_try_allocate_for_objc):
(iso_has_object):
(iso_get_allocation_size):
(iso_get_heap):
(iso_deallocate):
(iso_force_primitive_heap_into_reserved_memory):
* libpas/src/libpas/iso_heap.h: Added.
* libpas/src/libpas/iso_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap_config.h: Added.
* libpas/src/libpas/iso_heap_inlines.h: Added.
(iso_try_allocate_common_primitive_inline):
(iso_try_allocate_common_primitive_with_alignment_inline):
(iso_try_allocate_common_primitive_zeroed_inline):
(iso_allocate_common_primitive_inline):
(iso_allocate_common_primitive_with_alignment_inline):
(iso_allocate_common_primitive_zeroed_inline):
(iso_try_reallocate_common_primitive_inline):
(iso_reallocate_common_primitive_inline):
(iso_try_allocate_inline):
(iso_allocate_inline):
(iso_try_allocate_array_inline):
(iso_allocate_array_inline):
(iso_try_allocate_array_zeroed_inline):
(iso_allocate_array_zeroed_inline):
(iso_try_reallocate_array_inline):
(iso_reallocate_array_inline):
(iso_try_allocate_primitive_inline):
(iso_allocate_primitive_inline):
(iso_try_allocate_primitive_zeroed_inline):
(iso_allocate_primitive_zeroed_inline):
(iso_try_allocate_primitive_with_alignment_inline):
(iso_allocate_primitive_with_alignment_inline):
(iso_try_reallocate_primitive_inline):
(iso_reallocate_primitive_inline):
(iso_try_allocate_for_objc_inline):
(iso_has_object_inline):
(iso_get_allocation_size_inline):
(iso_get_heap_inline):
(iso_deallocate_inline):
* libpas/src/libpas/iso_heap_innards.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_heap_ref.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap.c: Added.
(iso_test_allocate_common_primitive):
(iso_test_allocate):
(iso_test_allocate_array):
(iso_test_deallocate):
(iso_test_heap_ref_get_heap):
* libpas/src/libpas/iso_test_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/iso_test_heap_config.h: Added.
* libpas/src/libpas/jit_heap.c: Added.
(jit_heap_add_fresh_memory):
(jit_heap_try_allocate):
(jit_heap_shrink):
(jit_heap_get_size):
(jit_heap_deallocate):
* libpas/src/libpas/jit_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/jit_heap_config.c: Added.
(fresh_memory_aligned_allocator):
(initialize_fresh_memory_config):
(allocate_from_fresh):
(page_provider):
(jit_page_header_for_boundary_remote):
(jit_small_bitfit_allocate_page):
(jit_small_bitfit_create_page_header):
(jit_small_bitfit_destroy_page_header):
(jit_medium_bitfit_allocate_page):
(jit_medium_bitfit_create_page_header):
(jit_medium_bitfit_destroy_page_header):
(jit_aligned_allocator):
(jit_prepare_to_enumerate):
(jit_heap_config_for_each_shared_page_directory):
(jit_heap_config_for_each_shared_page_directory_remote):
(jit_heap_config_add_fresh_memory):
* libpas/src/libpas/jit_heap_config.h: Added.
(jit_type_size):
(jit_type_alignment):
(jit_heap_config_fast_megapage_kind):
(jit_small_bitfit_page_header_for_boundary):
(jit_small_bitfit_boundary_for_page_header):
(jit_medium_bitfit_page_header_for_boundary):
(jit_medium_bitfit_boundary_for_page_header):
(jit_heap_config_page_header):
* libpas/src/libpas/jit_heap_config_root_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/minalign32_heap.c: Added.
(minalign32_allocate_common_primitive):
(minalign32_allocate):
(minalign32_allocate_array):
(minalign32_deallocate):
(minalign32_heap_ref_get_heap):
* libpas/src/libpas/minalign32_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/minalign32_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(minalign32_heap_config_activate):
* libpas/src/libpas/minalign32_heap_config.h: Added.
* libpas/src/libpas/pagesize64k_heap.c: Added.
(pagesize64k_allocate_common_primitive):
(pagesize64k_allocate):
(pagesize64k_allocate_array):
(pagesize64k_deallocate):
(pagesize64k_heap_ref_get_heap):
* libpas/src/libpas/pagesize64k_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pagesize64k_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(pagesize64k_heap_config_activate):
* libpas/src/libpas/pagesize64k_heap_config.h: Added.
* libpas/src/libpas/pas_aligned_allocation_result.h: Added.
(pas_aligned_allocation_result_create_empty):
(pas_aligned_allocation_result_as_allocation_result):
* libpas/src/libpas/pas_aligned_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_alignment.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_alignment_dump):
* libpas/src/libpas/pas_alignment.h: Added.
(pas_alignment_create):
(pas_alignment_create_traditional):
(pas_alignment_create_trivial):
(pas_alignment_validate):
(pas_alignment_is_ptr_aligned):
(pas_alignment_round_up):
(pas_alignment_is_equal):
* libpas/src/libpas/pas_all_biasing_directories.c: Added.
(pas_all_biasing_directories_append):
(pas_all_biasing_directories_activate):
(scavenge_for_each_set_bit_bits_source):
(scavenge_for_each_set_bit_callback):
(scavenge_bitvector_word_callback):
(pas_all_biasing_directories_scavenge):
* libpas/src/libpas/pas_all_biasing_directories.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_heap_configs.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_heaps.c: Added.
(pas_all_heaps_add_heap):
(pas_all_heaps_for_each_static_heap):
(pas_all_heaps_for_each_dynamic_heap):
(pas_all_heaps_for_each_heap):
(pas_all_heaps_for_each_static_segregated_heap_not_part_of_a_heap):
(for_each_segregated_heap_callback):
(pas_all_heaps_for_each_static_segregated_heap):
(pas_all_heaps_for_each_segregated_heap):
(get_num_free_bytes_for_each_heap_callback):
(pas_all_heaps_get_num_free_bytes):
(reset_heap_ref_for_each_heap_callback):
(pas_all_heaps_reset_heap_ref):
(for_each_segregated_directory_global_size_directory_callback):
(for_each_segregated_directory_shared_page_directory_callback):
(for_each_segregated_directory_segregated_heap_callback):
(pas_all_heaps_for_each_segregated_directory):
(dump_directory_nicely):
(dump_view_nicely):
(verify_in_steady_state_segregated_directory_callback):
(pas_all_heaps_verify_in_steady_state):
(compute_total_non_utility_segregated_summary_directory_callback):
(pas_all_heaps_compute_total_non_utility_segregated_summary):
(compute_total_non_utility_bitfit_summary_heap_callback):
(pas_all_heaps_compute_total_non_utility_bitfit_summary):
(compute_total_non_utility_large_summary_heap_callback):
(pas_all_heaps_compute_total_non_utility_large_summary):
(pas_all_heaps_compute_total_non_utility_summary):
* libpas/src/libpas/pas_all_heaps.h: Added.
* libpas/src/libpas/pas_all_magazines.c: Added.
(pas_all_magazines_get_current):
* libpas/src/libpas/pas_all_magazines.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_all_shared_page_directories.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_all_shared_page_directories_add):
(pas_all_shared_page_directories_for_each):
* libpas/src/libpas/pas_all_shared_page_directories.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_callbacks.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_callbacks.h: Added.
(pas_did_allocate):
(pas_will_deallocate):
* libpas/src/libpas/pas_allocation_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocation_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocation_kind_get_string):
* libpas/src/libpas/pas_allocation_result.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocation_result_zero):
* libpas/src/libpas/pas_allocation_result.h: Added.
(pas_allocation_result_create_failure):
(pas_allocation_result_create_success_with_zero_mode):
(pas_allocation_result_create_success):
* libpas/src/libpas/pas_allocator_counts.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocator_index.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_allocator_scavenge_action.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_allocator_scavenge_action_get_string):
* libpas/src/libpas/pas_baseline_allocator.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_baseline_allocator_attach_directory):
(pas_baseline_allocator_detach_directory):
* libpas/src/libpas/pas_baseline_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_baseline_allocator_result.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_baseline_allocator_result_create_failure):
(pas_baseline_allocator_result_create_success):
* libpas/src/libpas/pas_baseline_allocator_table.c: Added.
(initialize):
(pas_baseline_allocator_table_initialize_if_necessary):
(pas_baseline_allocator_table_get_random_index):
(pas_baseline_allocator_table_for_all):
* libpas/src/libpas/pas_baseline_allocator_table.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_config_enumerator_data.c: Added.
(pas_basic_heap_config_enumerator_data_add_page_header_table):
* libpas/src/libpas/pas_basic_heap_config_enumerator_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_config_root_data.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_heap_page_caches.h: Added.
(pas_basic_heap_page_caches_get_shared_page_directories):
* libpas/src/libpas/pas_basic_heap_runtime_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_basic_segregated_page_caches.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_biasing_directory.c: Added.
(pas_biasing_directory_construct):
(pas_biasing_directory_get_sharing_pool):
(pas_biasing_directory_did_create_delta):
(pas_biasing_directory_take_last_unused):
(pas_biasing_directory_did_use_index_slow):
(pas_biasing_directory_index_did_become_eligible):
* libpas/src/libpas/pas_biasing_directory.h: Added.
(pas_is_segregated_biasing_directory):
(pas_is_bitfit_biasing_directory):
(pas_biasing_directory_magazine_index):
(pas_biasing_directory_ownership_threshold):
(pas_biasing_directory_unused_span_size):
(pas_biasing_directory_did_use_index):
* libpas/src/libpas/pas_biasing_directory_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_unwrap_segregated_biasing_directory):
(pas_unwrap_bitfit_biasing_directory):
* libpas/src/libpas/pas_biasing_directory_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_biasing_directory_kind_get_string):
* libpas/src/libpas/pas_bitfield_vector.h: Added.
(pas_bitfield_vector_get):
(pas_bitfield_vector_set):
* libpas/src/libpas/pas_bitfit_allocation_result.h: Added.
(pas_bitfit_allocation_result_create_success):
(pas_bitfit_allocation_result_create_failure):
(pas_bitfit_allocation_result_create_empty):
(pas_bitfit_allocation_result_create_need_to_lock_commit_lock):
* libpas/src/libpas/pas_bitfit_allocator.c: Added.
(pas_bitfit_allocator_commit_view):
(pas_bitfit_allocator_finish_failing):
* libpas/src/libpas/pas_bitfit_allocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_allocator_inlines.h: Added.
(pas_bitfit_allocator_reset):
(pas_bitfit_allocator_assert_reset):
(pas_bitfit_allocator_try_allocate):
* libpas/src/libpas/pas_bitfit_biasing_directory.c: Added.
(pas_bitfit_biasing_directory_create):
(pas_bitfit_biasing_directory_take_last_unused):
* libpas/src/libpas/pas_bitfit_biasing_directory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_directory.c: Added.
(pas_bitfit_directory_construct):
(pas_bitfit_directory_update_biasing_eligibility):
(pas_bitfit_directory_max_free_did_become_unprocessed):
(pas_bitfit_directory_max_free_did_become_unprocessed_unchecked):
(pas_bitfit_directory_max_free_did_become_empty_without_biasing_update):
(pas_bitfit_directory_max_free_did_become_empty):
(pas_bitfit_directory_get_first_free_view):
(pas_bitfit_directory_compute_summary):
(for_each_live_object_callback):
(pas_bitfit_directory_for_each_live_object):
* libpas/src/libpas/pas_bitfit_directory.h: Added.
(pas_bitfit_directory_size):
(pas_bitfit_directory_get_max_free_ptr):
(pas_bitfit_directory_get_max_free):
(pas_bitfit_directory_set_max_free_unchecked):
(pas_bitfit_directory_set_max_free_not_empty_impl):
(pas_bitfit_directory_set_processed_max_free):
(pas_bitfit_directory_set_unprocessed_max_free_unchecked):
(pas_bitfit_directory_set_empty_max_free):
(pas_bitfit_directory_set_unprocessed_max_free):
(pas_bitfit_directory_get_view_ptr):
(pas_bitfit_directory_get_view):
* libpas/src/libpas/pas_bitfit_directory_and_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_directory_and_index_create):
* libpas/src/libpas/pas_bitfit_directory_inlines.h: Added.
(pas_bitfit_directory_get_global):
(pas_bitfit_directory_find_first_free_for_num_bits_iterate_callback):
(pas_bitfit_directory_find_first_free_for_num_bits):
(pas_bitfit_directory_find_first_free):
(pas_bitfit_directory_find_first_empty_iterate_callback):
(pas_bitfit_directory_find_first_empty):
* libpas/src/libpas/pas_bitfit_directory_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_directory_kind_get_string):
* libpas/src/libpas/pas_bitfit_global_directory.c: Added.
(pas_bitfit_global_directory_construct):
(pas_bitfit_global_directory_does_sharing):
(pas_bitfit_global_directory_get_use_epoch):
(pas_bitfit_global_directory_get_empty_bit_at_index):
(pas_bitfit_global_directory_set_empty_bit_at_index):
(pas_bitfit_global_directory_view_did_become_empty_at_index):
(pas_bitfit_global_directory_view_did_become_empty):
(pas_bitfit_global_directory_take_last_empty):
(pas_bitfit_global_directory_dump_reference):
(pas_bitfit_global_directory_dump_for_spectrum):
* libpas/src/libpas/pas_bitfit_global_directory.h: Added.
* libpas/src/libpas/pas_bitfit_global_size_class.c: Added.
(pas_bitfit_global_size_class_create):
(pas_bitfit_global_size_class_select_for_magazine):
* libpas/src/libpas/pas_bitfit_global_size_class.h: Added.
* libpas/src/libpas/pas_bitfit_heap.c: Added.
(pas_bitfit_heap_create):
(pas_bitfit_heap_select_variant):
(pas_bitfit_heap_ensure_global_size_class):
(pas_bitfit_heap_compute_summary):
(for_each_live_object_callback):
(pas_bitfit_heap_for_each_live_object):
* libpas/src/libpas/pas_bitfit_heap.h: Added.
(pas_bitfit_heap_get_directory):
* libpas/src/libpas/pas_bitfit_max_free.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page.c: Added.
(pas_bitfit_page_construct):
(pas_bitfit_page_get_config):
(pas_bitfit_page_log_bits):
(pas_bitfit_page_deallocation_did_fail):
(pas_bitfit_page_for_each_live_object):
(verify_for_each_object_callback):
(pas_bitfit_page_verify):
* libpas/src/libpas/pas_bitfit_page.h: Added.
(pas_bitfit_page_header_size):
(pas_bitfit_page_free_bits):
(pas_bitfit_page_object_end_bits):
(pas_bitfit_page_get_granule_use_counts):
(pas_bitfit_page_offset_to_first_object):
(pas_bitfit_page_offset_to_end_of_last_object):
(pas_bitfit_page_payload_size):
(pas_bitfit_page_for_boundary):
(pas_bitfit_page_for_boundary_or_null):
(pas_bitfit_page_for_boundary_unchecked):
(pas_bitfit_page_boundary):
(pas_bitfit_page_boundary_or_null):
(pas_bitfit_page_for_address_and_page_config):
(pas_bitfit_page_testing_verify):
* libpas/src/libpas/pas_bitfit_page_config.h: Added.
(pas_bitfit_page_config_is_enabled):
(pas_bitfit_page_config_num_alloc_bits):
(pas_bitfit_page_config_num_alloc_words):
(pas_bitfit_page_config_num_alloc_words64):
(pas_bitfit_page_config_num_alloc_bit_bytes):
(pas_bitfit_page_config_byte_offset_for_object_bits):
(pas_bitfit_page_config_uses_subpages):
* libpas/src/libpas/pas_bitfit_page_config_inlines.h: Added.
* libpas/src/libpas/pas_bitfit_page_config_kind.c: Added.
(pas_bitfit_page_config_kind_for_each):
* libpas/src/libpas/pas_bitfit_page_config_kind.def: Added.
* libpas/src/libpas/pas_bitfit_page_config_kind.h: Added.
(pas_bitfit_page_config_kind_get_string):
(pas_bitfit_page_config_kind_get_config):
* libpas/src/libpas/pas_bitfit_page_config_utils.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page_config_utils_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bitfit_page_config_variant.h: Added.
(pas_bitfit_page_config_variant_get_string):
(pas_bitfit_page_config_variant_get_capitalized_string):
* libpas/src/libpas/pas_bitfit_page_inlines.h: Added.
(pas_bitfit_page_compute_offset):
(pas_bitfit_page_allocation_satisfies_alignment):
(pas_bitfit_page_allocation_commit_granules_or_reloop):
(pas_bitfit_page_finish_allocation):
(pas_bitfit_page_allocate):
(pas_bitfit_page_deallocate_with_page_impl_mode_get_string):
(pas_bitfit_page_deallocate_with_page_impl):
(pas_bitfit_page_deallocate_with_page):
(pas_bitfit_page_deallocate):
(pas_bitfit_page_get_allocation_size_with_page):
(pas_bitfit_page_get_allocation_size):
(pas_bitfit_page_shrink_with_page):
(pas_bitfit_page_shrink):
* libpas/src/libpas/pas_bitfit_size_class.c: Added.
(pas_bitfit_size_class_find_insertion_point):
(pas_bitfit_size_class_construct):
(pas_bitfit_size_class_create):
(pas_bitfit_size_class_get_first_free_view):
* libpas/src/libpas/pas_bitfit_size_class.h: Added.
* libpas/src/libpas/pas_bitfit_view.c: Added.
(pas_bitfit_view_create):
(pas_bitfit_view_lock_ownership_lock_slow):
(pas_bitfit_view_note_nonemptiness):
(did_become_empty_for_bits):
(pas_bitfit_view_note_full_emptiness):
(pas_bitfit_view_note_partial_emptiness):
(pas_bitfit_view_note_max_free):
(compute_summary):
(pas_bitfit_view_compute_summary):
(for_each_live_object_callback):
(for_each_live_object):
(pas_bitfit_view_for_each_live_object):
* libpas/src/libpas/pas_bitfit_view.h: Added.
(pas_bitfit_view_lock_ownership_lock):
* libpas/src/libpas/pas_bitfit_view_and_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bitfit_view_and_index_create):
(pas_bitfit_view_and_index_create_empty):
* libpas/src/libpas/pas_bitfit_view_inlines.h: Added.
(pas_bitfit_view_current_directory_and_index):
(pas_bitfit_view_current_directory):
(pas_bitfit_view_index_in_current):
(pas_bitfit_view_is_empty):
* libpas/src/libpas/pas_bitvector.h: Added.
(pas_bitvector_get_from_word):
(pas_bitvector_get):
(pas_bitvector_get_from_one_word):
(pas_backward_bitvector_get):
(pas_bitvector_set_in_word):
(pas_bitvector_set):
(pas_bitvector_set_in_one_word):
(pas_bitvector_set_atomic_in_word):
(pas_bitvector_set_atomic):
(pas_backward_bitvector_set):
(pas_bitvector_find_first_set):
(pas_bitvector_for_each_set_bit):
(pas_bitvector64_set_range):
* libpas/src/libpas/pas_bootstrap_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(bootstrap_source_allocate_aligned):
(initialize_config):
* libpas/src/libpas/pas_bootstrap_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_bootstrap_heap_page_provider.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_bootstrap_heap_page_provider):
* libpas/src/libpas/pas_bootstrap_heap_page_provider.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_cares_about_size_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_cartesian_tree.h: Added.
(pas_cartesian_tree_node_child_ptr):
(pas_cartesian_tree_node_is_null_constrained):
(pas_cartesian_tree_node_minimum):
(pas_cartesian_tree_node_minimum_constrained):
(pas_cartesian_tree_node_maximum):
(pas_cartesian_tree_node_maximum_constrained):
(pas_cartesian_tree_node_successor):
(pas_cartesian_tree_node_successor_constrained):
(pas_cartesian_tree_node_predecessor):
(pas_cartesian_tree_node_predecessor_constrained):
(pas_cartesian_tree_node_reset):
(pas_cartesian_tree_construct):
(pas_cartesian_tree_node_find_exact):
(pas_cartesian_tree_find_exact):
(pas_cartesian_tree_node_find_least_greater_than_or_equal):
(pas_cartesian_tree_find_least_greater_than_or_equal):
(pas_cartesian_tree_node_find_least_greater_than):
(pas_cartesian_tree_find_least_greater_than):
(pas_cartesian_tree_node_find_greatest_less_than_or_equal):
(pas_cartesian_tree_find_greatest_less_than_or_equal):
(pas_cartesian_tree_node_find_greatest_less_than):
(pas_cartesian_tree_find_greatest_less_than):
(pas_cartesian_tree_minimum):
(pas_cartesian_tree_minimum_constrained):
(pas_cartesian_tree_maximum):
(pas_cartesian_tree_maximum_constrained):
(pas_cartesian_tree_is_empty):
(pas_cartesian_tree_insert):
(pas_cartesian_tree_remove):
(pas_cartesian_tree_validate_recurse):
(pas_cartesian_tree_validate):
(pas_cartesian_tree_size):
* libpas/src/libpas/pas_coalign.c: Added.
(formal_mod):
(ceiling_div):
(pas_coalign_one_sided):
(pas_coalign):
* libpas/src/libpas/pas_coalign.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_coalign_empty_result):
(pas_coalign_result_create):
* libpas/src/libpas/pas_commit_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_commit_mode_get_string):
* libpas/src/libpas/pas_commit_span.c: Added.
(pas_commit_span_construct):
(pas_commit_span_add_to_change):
(pas_commit_span_add_unchanged):
(commit):
(pas_commit_span_add_unchanged_and_commit):
(decommit):
(pas_commit_span_add_unchanged_and_decommit):
* libpas/src/libpas/pas_commit_span.h: Added.
* libpas/src/libpas/pas_compact_atomic_allocator_index_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_global_size_class_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_size_class_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_bitfit_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_enumerable_range_list_chunk_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_page_sharing_pool_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_ptr.h: Added.
* libpas/src/libpas/pas_compact_atomic_segregated_exclusive_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_global_size_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_heap_page_sharing_pools_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_partial_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_segregated_view.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_thread_local_cache_layout_node.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_atomic_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_global_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bitfit_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_bootstrap_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
(compact_bootstrap_source_allocate_aligned):
(initialize_config):
* libpas/src/libpas/pas_compact_bootstrap_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_cartesian_tree_node_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_heap_reservation.c: Added.
(pas_compact_heap_reservation_try_allocate):
* libpas/src/libpas/pas_compact_heap_reservation.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_large_utility_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_large_utility_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_page_granule_use_count_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_ptr.h: Added.
* libpas/src/libpas/pas_compact_segregated_biasing_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_global_size_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_heap_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_shared_page_directory_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_shared_view_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_segregated_view.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_skip_list_node_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_skip_list_node_ptr_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_subpage_map_entry_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_tagged_atomic_ptr.h: Added.
* libpas/src/libpas/pas_compact_tagged_page_granule_use_count_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_tagged_ptr.h: Added.
* libpas/src/libpas/pas_compact_tagged_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compact_unsigned_ptr.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_compute_summary_object_callbacks.c: Added.
(pas_compute_summary_live_object_callback):
(pas_compute_summary_live_object_callback_without_physical_sharing):
(config):
(pas_compute_summary_dead_object_callback):
(pas_compute_summary_dead_object_callback_without_physical_sharing):
* libpas/src/libpas/pas_compute_summary_object_callbacks.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_config.h: Added.
* libpas/src/libpas/pas_config_prefix.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_count_lookup_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_count_lookup_mode_get_string):
* libpas/src/libpas/pas_create_basic_heap_page_caches_with_reserved_memory.c: Added.
(pas_create_basic_heap_page_caches_with_reserved_memory):
* libpas/src/libpas/pas_create_basic_heap_page_caches_with_reserved_memory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deallocate.c: Added.
(pas_try_deallocate_known_large):
(pas_deallocate_known_large):
(pas_try_deallocate_slow):
(deallocate_segregated):
(pas_try_deallocate_slow_no_cache):
* libpas/src/libpas/pas_deallocate.h: Added.
(pas_deallocate_known_segregated_impl):
(pas_deallocate_known_segregated):
(pas_try_deallocate_not_small):
(pas_try_deallocate_impl):
(pas_try_deallocate):
(pas_deallocate):
* libpas/src/libpas/pas_deallocation_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_deallocation_mode_get_string):
* libpas/src/libpas/pas_deallocator.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deallocator_scavenge_action.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_deallocator_scavenge_action_get_string):
* libpas/src/libpas/pas_debug_heap.h: Added.
(pas_debug_heap_is_enabled):
(pas_debug_heap_malloc):
(pas_debug_heap_memalign):
(pas_debug_heap_realloc):
(pas_debug_heap_free):
* libpas/src/libpas/pas_debug_spectrum.c: Added.
(pas_debug_spectrum_add):
(pas_debug_spectrum_dump):
(pas_debug_spectrum_reset):
* libpas/src/libpas/pas_debug_spectrum.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_deferred_decommit_log.c: Added.
(pas_deferred_decommit_log_construct):
(pas_deferred_decommit_log_destruct):
(already_holds_lock):
(pas_deferred_decommit_log_lock_for_adding):
(pas_deferred_decommit_log_add):
(pas_deferred_decommit_log_add_already_locked):
(pas_deferred_decommit_log_add_maybe_locked):
(pas_deferred_decommit_log_unlock_after_aborted_add):
(decommit_all):
(pas_deferred_decommit_log_decommit_all):
(pas_deferred_decommit_log_pretend_to_decommit_all):
* libpas/src/libpas/pas_deferred_decommit_log.h: Added.
* libpas/src/libpas/pas_designated_intrinsic_heap.c: Added.
(set_up_range):
(pas_designated_intrinsic_heap_initialize):
* libpas/src/libpas/pas_designated_intrinsic_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_designated_intrinsic_heap_inlines.h: Added.
(pas_designated_intrinsic_heap_num_allocator_indices):
(pas_designated_index_result_create_failure):
(pas_designated_index_result_create_success):
(pas_designated_intrinsic_heap_num_designated_indices):
(pas_designated_intrinsic_heap_designated_index):
* libpas/src/libpas/pas_dyld_state.c: Added.
(pas_dyld_is_libsystem_initialized):
* libpas/src/libpas/pas_dyld_state.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_dynamic_primitive_heap_map.c: Added.
(pas_dynamic_primitive_heap_map_find_slow):
* libpas/src/libpas/pas_dynamic_primitive_heap_map.h: Added.
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_create_empty):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_create_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_empty_or_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_empty):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_is_deleted):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_entry_get_key):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_key_get_hash):
(pas_dynamic_primitive_heap_map_heaps_for_size_table_key_is_equal):
(pas_dynamic_primitive_heap_map_hash):
(pas_dynamic_primitive_heap_map_find):
* libpas/src/libpas/pas_ensure_heap_forced_into_reserved_memory.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_forced_into_reserved_memory):
* libpas/src/libpas/pas_ensure_heap_forced_into_reserved_memory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_ensure_heap_with_page_caches.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_with_page_caches):
* libpas/src/libpas/pas_ensure_heap_with_page_caches.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerable_page_malloc.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_enumerable_page_malloc_try_allocate_without_deallocating_padding):
* libpas/src/libpas/pas_enumerable_page_malloc.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerable_range_list.c: Added.
(pas_enumerable_range_list_append):
(pas_enumerable_range_list_iterate):
(pas_enumerable_range_list_iterate_remote):
* libpas/src/libpas/pas_enumerable_range_list.h: Added.
* libpas/src/libpas/pas_enumerate_bitfit_heaps.c: Added.
(view_callback):
(enumerate_bitfit_directory):
(enumerate_bitfit_heap_callback):
(pas_enumerate_bitfit_heaps):
* libpas/src/libpas/pas_enumerate_bitfit_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_initially_unaccounted_pages.c: Added.
(range_list_iterate_add_unaccounted_callback):
(range_list_iterate_exclude_accounted_callback):
(pas_enumerate_initially_unaccounted_pages):
* libpas/src/libpas/pas_enumerate_initially_unaccounted_pages.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_large_heaps.c: Added.
(range_list_iterate_add_large_payload_callback):
(record_span):
(large_map_hashtable_entry_callback):
(small_large_map_hashtable_entry_callback):
(tiny_large_map_second_level_hashtable_entry_callback):
(tiny_large_map_hashtable_entry_callback):
(pas_enumerate_large_heaps):
* libpas/src/libpas/pas_enumerate_large_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_segregated_heaps.c: Added.
(local_allocator_map_entry_create_empty):
(local_allocator_map_entry_create_deleted):
(local_allocator_map_entry_is_empty_or_deleted):
(local_allocator_map_entry_is_empty):
(local_allocator_map_entry_is_deleted):
(local_allocator_map_entry_get_key):
(local_allocator_map_key_get_hash):
(local_allocator_map_key_is_equal):
(for_each_view):
(collect_shared_page_directories_shared_page_directory_callback):
(collect_shared_page_directories_heap_callback):
(record_page_payload_and_meta):
(record_page_objects):
(enumerate_exclusive_view):
(enumerate_shared_view):
(enumerate_partial_view):
(shared_page_directory_view_callback):
(size_directory_view_callback):
(enumerate_segregated_heap_callback):
(consider_allocator):
(pas_enumerate_segregated_heaps):
* libpas/src/libpas/pas_enumerate_segregated_heaps.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerate_unaccounted_pages_as_meta.c: Added.
(pas_enumerate_unaccounted_pages_as_meta):
* libpas/src/libpas/pas_enumerate_unaccounted_pages_as_meta.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_enumerator.c: Added.
(allocate):
(deallocate):
(pas_enumerator_create):
(pas_enumerator_destroy):
(pas_enumerator_allocate):
(pas_enumerator_read_compact):
(pas_enumerator_read):
(pas_enumerator_add_unaccounted_pages):
(pas_enumerator_exclude_accounted_page):
(pas_enumerator_exclude_accounted_pages):
(pas_enumerator_record):
(record_payload_span):
(pas_enumerator_record_page_payload_and_meta):
(pas_enumerator_for_each_heap):
(pas_enumerator_enumerate_all):
* libpas/src/libpas/pas_enumerator.h: Added.
(pas_enumerator_record_kind_get_string):
* libpas/src/libpas/pas_enumerator_internal.h: Added.
* libpas/src/libpas/pas_enumerator_region.c: Added.
(pas_enumerator_region_allocate):
(pas_enumerator_region_destroy):
* libpas/src/libpas/pas_enumerator_region.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_epoch.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_get_epoch):
* libpas/src/libpas/pas_epoch.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_exclusive_view_template_memo_table.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_exclusive_view_template_memo_table.h: Added.
(pas_exclusive_view_template_memo_key_create):
(pas_exclusive_view_template_memo_entry_create_empty):
(pas_exclusive_view_template_memo_entry_create_deleted):
(pas_exclusive_view_template_memo_entry_is_empty_or_deleted):
(pas_exclusive_view_template_memo_entry_is_empty):
(pas_exclusive_view_template_memo_entry_is_deleted):
(pas_exclusive_view_template_memo_entry_get_key):
(pas_exclusive_view_template_memo_key_get_hash):
(pas_exclusive_view_template_memo_key_is_equal):
* libpas/src/libpas/pas_extended_gcd.c: Added.
(pas_extended_gcd):
* libpas/src/libpas/pas_extended_gcd.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_fast_large_free_heap.c: Added.
(key_compare_callback):
(get_x_key_callback):
(get_y_key_callback):
(initialize_cartesian_config):
(pas_fast_large_free_heap_construct):
(insert_node):
(remove_node):
(dump_heap):
(fast_find_first):
(fast_find_by_end):
(fast_read_cursor):
(fast_write_cursor):
(fast_merge):
(fast_remove):
(fast_append):
(fast_commit):
(fast_dump):
(fast_add_mapped_bytes):
(initialize_generic_heap_config):
(pas_fast_large_free_heap_try_allocate):
(pas_fast_large_free_heap_deallocate):
(pas_fast_large_free_heap_for_each_free):
(pas_fast_large_free_heap_get_num_free_bytes):
(pas_fast_large_free_heap_validate):
(pas_fast_large_free_heap_dump_to_printf):
* libpas/src/libpas/pas_fast_large_free_heap.h: Added.
(pas_fast_large_free_heap_get_num_mapped_bytes):
* libpas/src/libpas/pas_fast_megapage_cache.c: Added.
(table_set_by_index):
(pas_fast_megapage_cache_try_allocate):
* libpas/src/libpas/pas_fast_megapage_cache.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_fast_megapage_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_fast_megapage_kind_get_string):
* libpas/src/libpas/pas_fast_megapage_table.c: Added.
(pas_fast_megapage_table_initialize_static):
(pas_fast_megapage_table_set_by_index):
* libpas/src/libpas/pas_fast_megapage_table.h: Added.
(pas_fast_megapage_table_get_by_index):
(pas_fast_megapage_table_get):
* libpas/src/libpas/pas_fast_path_allocation_result.h: Added.
(pas_fast_path_allocation_result_create):
(pas_fast_path_allocation_result_create_need_slow):
(pas_fast_path_allocation_result_create_out_of_memory):
(pas_fast_path_allocation_result_create_success):
(pas_fast_path_allocation_result_from_allocation_result):
(pas_fast_path_allocation_result_to_allocation_result):
* libpas/src/libpas/pas_fast_path_allocation_result_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_fast_path_allocation_result_kind_get_string):
* libpas/src/libpas/pas_fast_tls.h: Added.
* libpas/src/libpas/pas_fd_stream.c: Copied from Source/WTF/wtf/FastTLS.h.
(fd_stream_vprintf):
(pas_fd_stream_vprintf):
* libpas/src/libpas/pas_fd_stream.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_first_level_tiny_large_map_entry.h: Added.
(pas_first_level_tiny_large_map_entry_create_empty):
(pas_first_level_tiny_large_map_entry_create_deleted):
(pas_first_level_tiny_large_map_entry_is_empty_or_deleted):
(pas_first_level_tiny_large_map_entry_is_empty):
(pas_first_level_tiny_large_map_entry_is_deleted):
(pas_first_level_tiny_large_map_entry_get_key):
(pas_first_level_tiny_large_map_key_get_hash):
(pas_first_level_tiny_large_map_key_is_equal):
* libpas/src/libpas/pas_found_bit_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_found_bit_index_create_empty):
(pas_found_bit_index_create):
* libpas/src/libpas/pas_found_index.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_found_index_create_empty):
(pas_found_index_create):
(pas_found_index_create_unsuccessful):
* libpas/src/libpas/pas_free_granules.c: Added.
(pas_free_granules_compute_and_mark_decommitted):
(pas_free_granules_unmark_decommitted):
(pas_free_granules_decommit_after_locking_range):
* libpas/src/libpas/pas_free_granules.h: Added.
(pas_free_granules_is_free):
* libpas/src/libpas/pas_free_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_free_range_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_free_range_kind_get_string):
* libpas/src/libpas/pas_full_alloc_bits.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_full_alloc_bits_create_empty):
(pas_full_alloc_bits_create):
* libpas/src/libpas/pas_full_alloc_bits_inlines.h: Added.
(pas_full_alloc_bits_create_for_exclusive):
(pas_full_alloc_bits_create_for_partial):
(pas_full_alloc_bits_create_for_view_and_directory):
(pas_full_alloc_bits_create_for_view):
* libpas/src/libpas/pas_generic_large_free_heap.h: Added.
(pas_generic_large_free_heap_merge_physical):
(pas_generic_large_free_heap_try_allocate_test_allocation_candidate):
(pas_generic_large_free_heap_try_allocate):
* libpas/src/libpas/pas_get_allocation_size.h: Added.
(pas_get_allocation_size):
* libpas/src/libpas/pas_get_heap.h: Added.
(pas_get_heap_known_segregated):
(pas_get_heap):
* libpas/src/libpas/pas_get_object_kind.h: Added.
(pas_get_object_kind):
* libpas/src/libpas/pas_get_page_base.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_get_page_base):
* libpas/src/libpas/pas_has_object.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_has_object):
* libpas/src/libpas/pas_hashtable.h: Added.
* libpas/src/libpas/pas_heap.c: Added.
(pas_heap_create):
(pas_heap_get_type_size):
(pas_heap_get_num_free_bytes):
(for_each_live_object_small_object_callback):
(for_each_live_object_large_object_callback):
(pas_heap_for_each_live_object):
(pas_heap_compute_summary):
(pas_heap_reset_heap_ref):
(pas_heap_ensure_size_directory_for_count_slow):
* libpas/src/libpas/pas_heap.h: Added.
(pas_heap_for_large_heap):
(pas_heap_for_segregated_heap):
* libpas/src/libpas/pas_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_config_activate):
* libpas/src/libpas/pas_heap_config.h: Added.
(pas_heap_config_segregated_page_config_ptr_for_variant):
(pas_heap_config_bitfit_page_config_ptr_for_variant):
(pas_heap_config_segregated_page_config_for_variant):
(pas_heap_config_bitfit_page_config_for_variant):
* libpas/src/libpas/pas_heap_config_inlines.h: Added.
* libpas/src/libpas/pas_heap_config_kind.c: Added.
(pas_heap_config_kind_for_each):
* libpas/src/libpas/pas_heap_config_kind.def: Added.
* libpas/src/libpas/pas_heap_config_kind.h: Added.
(pas_heap_config_kind_get_string):
(pas_heap_config_kind_get_config):
(pas_heap_config_kind_is_active):
* libpas/src/libpas/pas_heap_config_utils.c: Added.
(pas_heap_config_utils_null_activate):
(pas_heap_config_utils_for_each_shared_page_directory):
(pas_heap_config_utils_for_each_shared_page_directory_remote):
(pas_heap_config_utils_allocate_aligned):
(pas_heap_config_utils_prepare_to_enumerate):
* libpas/src/libpas/pas_heap_config_utils.h: Added.
* libpas/src/libpas/pas_heap_config_utils_inlines.h: Added.
* libpas/src/libpas/pas_heap_for_config.c: Added.
(pas_heap_for_config_allocate):
(pas_heap_for_page_config_kind_allocate):
(pas_heap_for_page_config_allocate):
(pas_heap_for_config_allocate_with_alignment):
(pas_heap_for_page_config_allocate_with_alignment):
(pas_heap_for_config_allocate_with_manual_alignment):
(pas_heap_for_page_config_kind_allocate_with_manual_alignment):
(pas_heap_for_page_config_allocate_with_manual_alignment):
(pas_heap_for_config_deallocate):
(pas_heap_for_page_config_kind_deallocate):
(pas_heap_for_page_config_deallocate):
* libpas/src/libpas/pas_heap_for_config.h: Added.
* libpas/src/libpas/pas_heap_inlines.h: Added.
(pas_heap_ensure_size_directory_for_count):
* libpas/src/libpas/pas_heap_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_kind_get_string):
* libpas/src/libpas/pas_heap_lock.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_lock.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_page_provider.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_ref.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ensure_heap_slow):
* libpas/src/libpas/pas_heap_ref.h: Added.
(pas_ensure_heap):
* libpas/src/libpas/pas_heap_ref_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_ref_kind_get_string):
* libpas/src/libpas/pas_heap_ref_prefix.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_runtime_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_heap_summary.c: Added.
(pas_heap_summary_validate):
(pas_heap_summary_dump):
* libpas/src/libpas/pas_heap_summary.h: Added.
(pas_heap_summary_create_empty):
(pas_heap_summary_is_empty):
(pas_heap_summary_add):
(pas_heap_summary_committed_objects):
(pas_heap_summary_total):
(pas_heap_summary_fragmentation):
* libpas/src/libpas/pas_heap_table.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_table_try_allocate_index):
* libpas/src/libpas/pas_heap_table.h: Added.
(pas_heap_table_has_index):
(pas_heap_table_get_index):
* libpas/src/libpas/pas_heap_table_state.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_heap_table_state_get_string):
* libpas/src/libpas/pas_immortal_heap.c: Added.
(bump_is_ok):
(pas_immortal_heap_allocate_with_manual_alignment):
(pas_immortal_heap_allocate_with_alignment):
(pas_immortal_heap_allocate):
(pas_immortal_heap_hold_lock_and_allocate):
(pas_immortal_heap_allocate_with_heap_lock_hold_mode):
(pas_immortal_heap_allocate_with_alignment_and_heap_lock_hold_mode):
* libpas/src/libpas/pas_immortal_heap.h: Added.
* libpas/src/libpas/pas_immutable_vector.h: Added.
* libpas/src/libpas/pas_internal_config.h: Added.
* libpas/src/libpas/pas_intrinsic_allocation_result.h: Added.
(pas_intrinsic_allocation_result_create_empty):
(pas_intrinsic_allocation_result_create):
(pas_intrinsic_allocation_result_identity):
(pas_intrinsic_allocation_result_zero):
(pas_intrinsic_allocation_result_set_errno):
(pas_intrinsic_allocation_result_crash_on_error):
* libpas/src/libpas/pas_intrinsic_heap_support.h: Added.
* libpas/src/libpas/pas_large_free.h: Added.
(pas_large_free_create_empty):
(pas_large_free_is_empty):
(pas_large_free_size):
(pas_large_free_offset_in_type_at_end):
* libpas/src/libpas/pas_large_free_heap_config.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_free_heap_declarations.def: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_free_heap_deferred_commit_log.c: Added.
(pas_large_free_heap_deferred_commit_log_construct):
(pas_large_free_heap_deferred_commit_log_destruct):
(pas_large_free_heap_deferred_commit_log_add):
(dump_large_commit):
(commit):
(commit_all):
(pas_large_free_heap_deferred_commit_log_commit_all):
(pas_large_free_heap_deferred_commit_log_pretend_to_commit_all):
* libpas/src/libpas/pas_large_free_heap_deferred_commit_log.h: Added.
* libpas/src/libpas/pas_large_free_heap_definitions.def: Added.
* libpas/src/libpas/pas_large_free_heap_helpers.c: Added.
(large_utility_aligned_allocator):
(initialize_config):
(pas_large_free_heap_helpers_try_allocate_with_alignment):
(pas_large_free_heap_helpers_deallocate):
(pas_large_free_heap_helpers_compute_summary):
* libpas/src/libpas/pas_large_free_heap_helpers.h: Added.
* libpas/src/libpas/pas_large_free_inlines.h: Added.
(pas_large_free_create_merged):
(pas_large_free_split):
(pas_large_free_allocation_candidate):
(pas_large_free_usable_space):
(pas_large_allocation_result_create_empty):
(pas_large_free_allocate):
(pas_large_free_create_merged_for_sure):
(pas_large_free_can_merge):
(pas_large_allocation_result_as_allocation_result):
* libpas/src/libpas/pas_large_free_visitor.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_heap.c: Added.
(pas_large_heap_construct):
(aligned_allocator):
(initialize_config):
(pas_large_heap_try_allocate):
(pas_large_heap_try_deallocate):
(pas_large_heap_try_shrink):
(pas_large_heap_shove_into_free):
(for_each_live_object_entry_callback):
(pas_large_heap_for_each_live_object):
(pas_large_heap_for_object):
(pas_large_heap_get_num_free_bytes):
(compute_summary_live_object_callback):
(pas_large_heap_compute_summary):
* libpas/src/libpas/pas_large_heap.h: Added.
* libpas/src/libpas/pas_large_heap_physical_page_sharing_cache.c: Added.
(large_aligned_allocator):
(pas_large_heap_physical_page_sharing_cache_construct):
(pas_large_heap_physical_page_sharing_cache_try_allocate_with_alignment):
* libpas/src/libpas/pas_large_heap_physical_page_sharing_cache.h: Added.
* libpas/src/libpas/pas_large_map.c: Added.
(pas_large_map_find):
(pas_large_map_add):
(pas_large_map_take):
(pas_large_map_for_each_entry):
* libpas/src/libpas/pas_large_map.h: Added.
* libpas/src/libpas/pas_large_map_entry.h: Added.
(pas_large_map_entry_create_empty):
(pas_large_map_entry_create_deleted):
(pas_large_map_entry_is_empty_or_deleted):
(pas_large_map_entry_is_empty):
(pas_large_map_entry_is_deleted):
(pas_large_map_entry_get_key):
(pas_large_map_key_get_hash):
(pas_large_map_key_is_equal):
* libpas/src/libpas/pas_large_sharing_pool.c: Added.
(node_compare_callback):
(inner_key_compare_callback):
(update_min_epoch):
(validate_min_heap):
(validate_node):
(validate_node_if_asserting_aggressively):
(create_node):
(create_and_insert):
(boot_tree):
(destroy_node):
(remove_from_min_heap):
(remove_and_destroy):
(predecessor):
(successor):
(states_match):
(is_eligible):
(belongs_in_min_heap):
(update_min_heap):
(split_node_and_get_right_impl):
(split_node_and_get_right):
(split_node_and_get_left):
(merge_if_possible):
(node_containing):
(min_node_for_range):
(max_node_for_range):
(splat_live_bytes):
(should_do_commit_stuff_to):
(splat_command_get_string):
(splat_command_get_free_mode):
(dump_large_commit):
(try_splat_impl):
(try_splat):
(splat):
(pas_large_sharing_pool_boot_free):
(pas_large_sharing_pool_free):
(pas_large_sharing_pool_allocate_and_commit):
(pas_large_sharing_pool_decommit_least_recently_used):
(pas_large_sharing_pool_validate):
(pas_large_sharing_pool_compute_summary):
(pas_large_sharing_pool_for_each):
* libpas/src/libpas/pas_large_sharing_pool.h: Added.
(pas_large_sharing_node_heap_compare):
(pas_large_sharing_node_heap_get_index):
(pas_large_sharing_node_heap_set_index):
* libpas/src/libpas/pas_large_sharing_pool_epoch_update_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_large_sharing_pool_epoch_update_mode_get_string):
* libpas/src/libpas/pas_large_utility_free_heap.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_large_utility_free_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_line_word_config.h: Added.
(pas_line_word_config_count_low_zeroes_8_bit):
(pas_line_word_config_count_high_zeroes_8_bit):
(pas_line_word_config_count_low_zeroes_16_bit):
(pas_line_word_config_count_high_zeroes_16_bit):
(pas_line_word_config_count_low_zeroes_32_bit):
(pas_line_word_config_count_high_zeroes_32_bit):
(pas_line_word_config_count_low_zeroes_64_bit):
(pas_line_word_config_count_high_zeroes_64_bit):
(pas_line_word_config_construct):
* libpas/src/libpas/pas_list_direction.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_list_direction_get_string):
(pas_list_direction_invert):
* libpas/src/libpas/pas_local_allocator.c: Added.
(pas_local_allocator_construct):
(pas_local_allocator_destruct):
(pas_local_allocator_reset):
(pas_local_allocator_move):
(pas_local_allocator_refill_with_bitfit):
(pas_local_allocator_finish_refill_with_bitfit):
(stop_impl):
(pas_local_allocator_stop):
(pas_local_allocator_scavenge):
* libpas/src/libpas/pas_local_allocator.h: Added.
(pas_local_allocator_is_null):
(pas_local_allocator_is_active):
(pas_local_allocator_has_bitfit):
(pas_local_allocator_get_bitfit):
(pas_local_allocator_page_boundary):
(pas_local_allocator_alignment):
* libpas/src/libpas/pas_local_allocator_config_kind.h: Added.
(pas_local_allocator_config_kind_is_primordial_partial):
(pas_local_allocator_config_kind_is_bitfit):
(pas_local_allocator_config_kind_create_normal):
(pas_local_allocator_config_kind_create_primordial_partial):
(pas_local_allocator_config_kind_create_bitfit):
(pas_local_allocator_config_kind_get_segregated_page_config_kind):
(pas_local_allocator_config_kind_get_bitfit_page_config_kind):
(pas_local_allocator_config_kind_get_string):
* libpas/src/libpas/pas_local_allocator_inlines.h: Added.
(pas_local_allocator_reset_impl):
(pas_local_allocator_set_up_bump):
(pas_local_allocator_scan_bits_to_set_up_use_counts_bits_source):
(pas_local_allocator_scan_bits_to_set_up_use_counts_bit_callback):
(pas_local_allocator_scan_bits_to_set_up_free_bits):
(pas_local_allocator_set_up_free_bits):
(pas_local_allocator_make_bump):
(pas_local_allocator_prepare_to_allocate):
(pas_local_allocator_set_up_primordial_bump):
(pas_local_allocator_start_allocating_in_primordial_partial_view):
(pas_local_allocator_bless_primordial_partial_view_before_stopping):
(pas_local_allocator_try_allocate_in_primordial_partial_view):
(pas_local_allocator_refill_with_known_config):
(pas_local_allocator_return_memory_to_page_bits_source):
(pas_local_allocator_return_memory_to_page_set_bit_callback):
(pas_local_allocator_return_memory_to_page):
(pas_local_allocator_try_allocate_with_free_bits):
(pas_local_allocator_try_allocate_inline_cases):
(pas_local_allocator_try_allocate_small_segregated_slow_impl):
(pas_local_allocator_try_allocate_small_segregated_slow):
(pas_local_allocator_try_allocate_out_of_line_cases):
(pas_local_allocator_try_allocate_slow_impl):
(pas_local_allocator_try_allocate_slow):
(pas_local_allocator_try_allocate):
* libpas/src/libpas/pas_local_allocator_line.h: Added.
(pas_local_allocator_line_fits_without_shift):
(pas_local_allocator_line_decode):
(pas_local_allocator_line_encode):
(pas_local_allocator_line_remaining):
(pas_local_allocator_line_set_remaining):
(pas_local_allocator_line_payload_end_delta):
(pas_local_allocator_line_set_payload_end_delta):
* libpas/src/libpas/pas_local_allocator_refill_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_local_allocator_result.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_local_allocator_result_create_failure):
(pas_local_allocator_result_create_success):
* libpas/src/libpas/pas_lock.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_lock_lock_slow):
* libpas/src/libpas/pas_lock.h: Added.
(pas_lock_construct):
(pas_lock_construct_disabled):
(pas_lock_lock):
(pas_lock_try_lock):
(pas_lock_unlock):
(pas_lock_assert_held):
(pas_lock_testing_assert_held):
(pas_lock_lock_with_mode):
(pas_lock_switch_with_mode):
(pas_lock_switch):
(pas_lock_lock_conditionally):
(pas_lock_unlock_conditionally):
(pas_lock_for_switch_conditionally):
(pas_lock_switch_conditionally):
* libpas/src/libpas/pas_lock_free_read_hashtable.h: Added.
(if):
* libpas/src/libpas/pas_lock_free_read_ptr_ptr_hashtable.c: Added.
(pas_lock_free_read_ptr_ptr_hashtable_set):
* libpas/src/libpas/pas_lock_free_read_ptr_ptr_hashtable.h: Added.
(pas_lock_free_read_ptr_ptr_hashtable_find):
(pas_lock_free_read_ptr_ptr_hashtable_size):
* libpas/src/libpas/pas_log.c: Added.
(pas_vlog_fd):
(pas_log_fd):
(pas_vlog):
(pas_log):
(pas_start_crash_logging):
* libpas/src/libpas/pas_log.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_magazine.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_magazine_create):
* libpas/src/libpas/pas_magazine.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_medium_megapage_cache.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_medium_megapage_cache_try_allocate):
* libpas/src/libpas/pas_medium_megapage_cache.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_megapage_cache.c: Added.
(megapage_cache_allocate_aligned):
(pas_megapage_cache_construct):
(pas_megapage_cache_try_allocate):
* libpas/src/libpas/pas_megapage_cache.h: Added.
* libpas/src/libpas/pas_min_heap.h: Added.
* libpas/src/libpas/pas_monotonic_time.c: Copied from Source/WTF/wtf/FastTLS.h.
(get_timebase_info):
(pas_get_current_monotonic_time_nanoseconds):
* libpas/src/libpas/pas_monotonic_time.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_mutation_count.h: Added.
(pas_mutation_count_start_mutating):
(pas_mutation_count_stop_mutating):
(pas_mutation_count_is_mutating):
(pas_mutation_count_matches):
(pas_mutation_count_depend):
* libpas/src/libpas/pas_object_kind.h: Added.
(pas_object_kind_get_string):
(pas_object_kind_for_segregated_variant):
(pas_object_kind_for_bitfit_variant):
* libpas/src/libpas/pas_object_size_mode.h: Added.
(pas_object_size_mode_get_string):
* libpas/src/libpas/pas_page_base.c: Added.
(pas_page_base_get_config):
(pas_page_base_get_granule_use_counts):
(pas_page_base_compute_committed_when_owned):
(pas_page_base_is_empty):
(pas_page_base_add_free_range):
* libpas/src/libpas/pas_page_base.h: Added.
(pas_page_base_construct):
(pas_page_base_get_kind):
(pas_page_base_get_config_kind):
(pas_page_base_is_segregated):
(pas_page_base_get_segregated):
(pas_page_base_is_bitfit):
(pas_page_base_get_bitfit):
(pas_page_base_index_of_object_at_offset_from_page_boundary):
(pas_page_base_object_offset_from_page_boundary_at_index):
(pas_page_base_boundary):
(pas_page_base_boundary_or_null):
(pas_page_base_for_boundary):
(pas_page_base_for_boundary_or_null):
(pas_page_base_boundary_for_address_and_page_config):
(pas_page_base_for_address_and_page_config):
* libpas/src/libpas/pas_page_base_config.h: Added.
(pas_page_base_config_min_align):
(pas_page_base_config_object_payload_end_offset_from_boundary):
(pas_page_base_config_num_granule_bytes):
(pas_page_base_config_get_config_kind):
(pas_page_base_config_is_segregated):
(pas_page_base_config_is_bitfit):
(pas_page_base_config_get_segregated):
(pas_page_base_config_get_bitfit):
* libpas/src/libpas/pas_page_base_config_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_base_config_is_utility):
* libpas/src/libpas/pas_page_base_config_utils.h: Added.
* libpas/src/libpas/pas_page_base_config_utils_inlines.h: Added.
* libpas/src/libpas/pas_page_base_inlines.h: Added.
(pas_page_base_free_granule_uses_in_range_action):
(pas_page_base_free_granule_uses_in_range):
* libpas/src/libpas/pas_page_config_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_config_kind_get_string):
* libpas/src/libpas/pas_page_granule_use_count.h: Added.
(pas_page_granule_get_indices):
(pas_page_granule_for_each_use_in_range):
(pas_page_granule_use_count_increment):
(pas_page_granule_increment_uses_for_range):
* libpas/src/libpas/pas_page_header_placement_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_header_placement_mode_get_string):
* libpas/src/libpas/pas_page_header_table.c: Added.
(pas_page_header_table_add):
(pas_page_header_table_remove):
* libpas/src/libpas/pas_page_header_table.h: Added.
(pas_page_header_table_hash):
(pas_page_header_table_get_boundary_ptr):
(pas_page_header_table_get_boundary):
(pas_page_header_table_get_for_boundary):
(pas_page_header_table_get_for_address):
* libpas/src/libpas/pas_page_kind.h: Added.
(pas_page_kind_get_string):
(pas_page_kind_get_config_kind):
(pas_page_kind_get_segregated_variant):
(pas_page_kind_get_bitfit_variant):
* libpas/src/libpas/pas_page_malloc.c: Added.
(pas_page_malloc_alignment_slow):
(pas_page_malloc_alignment_shift_slow):
(pas_page_malloc_try_allocate_without_deallocating_padding):
(pas_page_malloc_commit):
(pas_page_malloc_decommit):
(pas_page_malloc_deallocate):
* libpas/src/libpas/pas_page_malloc.h: Added.
(pas_page_malloc_alignment):
(pas_page_malloc_alignment_shift):
* libpas/src/libpas/pas_page_sharing_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_sharing_mode_does_sharing):
(pas_page_sharing_mode_get_string):
* libpas/src/libpas/pas_page_sharing_participant.c: Added.
(pas_page_sharing_participant_create):
(pas_page_sharing_participant_get_payload):
(pas_page_sharing_participant_payload_construct):
(pas_page_sharing_participant_payload_with_use_epoch_construct):
(pas_page_sharing_participant_get_use_epoch):
(pas_page_sharing_participant_set_parent_pool):
(pas_page_sharing_participant_get_parent_pool):
(pas_page_sharing_participant_is_eligible):
(pas_page_sharing_participant_take_least_recently_used):
* libpas/src/libpas/pas_page_sharing_participant.h: Added.
(pas_page_sharing_participant_get_ptr):
(pas_page_sharing_participant_get_kind):
* libpas/src/libpas/pas_page_sharing_participant_kind.h: Added.
(pas_page_sharing_participant_kind_select_for_segregated_directory):
(pas_page_sharing_participant_kind_get_string):
* libpas/src/libpas/pas_page_sharing_pool.c: Added.
(verify_participants):
(verify_min_heap):
(dump_min_heap):
(pas_page_sharing_pool_construct):
(pas_page_sharing_pool_add_at_index):
(pas_page_sharing_pool_add):
(get_current_participant):
(take_from):
(pas_page_sharing_pool_take_least_recently_used):
(atomic_add_balance):
(pas_physical_page_sharing_pool_take):
(pas_physical_page_sharing_pool_scavenge):
(pas_physical_page_sharing_pool_take_later):
(pas_physical_page_sharing_pool_give_back):
(pas_physical_page_sharing_pool_take_for_page_config):
(pas_bias_page_sharing_pool_take):
(pas_page_sharing_pool_did_create_delta):
(pas_page_sharing_pool_verify):
(pas_page_sharing_pool_has_delta):
(pas_page_sharing_pool_has_current_participant):
* libpas/src/libpas/pas_page_sharing_pool.h: Added.
(pas_page_sharing_participant_compare):
(pas_page_sharing_participant_get_index):
(pas_page_sharing_participant_set_index):
(pas_page_sharing_pool_num_participants):
(pas_page_sharing_pool_get_participant):
* libpas/src/libpas/pas_page_sharing_pool_take_result.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_sharing_pool_take_result_get_string):
* libpas/src/libpas/pas_payload_reservation_page_list.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_payload_reservation_page_list_append):
* libpas/src/libpas/pas_payload_reservation_page_list.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_physical_memory_synchronization_style.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_physical_memory_synchronization_style_get_string):
* libpas/src/libpas/pas_physical_memory_transaction.c: Added.
(pas_physical_memory_transaction_construct):
(pas_physical_memory_transaction_begin):
(pas_physical_memory_transaction_end):
(pas_physical_memory_transaction_did_fail_to_acquire_lock):
* libpas/src/libpas/pas_physical_memory_transaction.h: Added.
* libpas/src/libpas/pas_primitive_heap_ref.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_primitive_heap_ref.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_promote_intrinsic_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_ptr_hash_map.h: Added.
(pas_ptr_hash_map_entry_create_empty):
(pas_ptr_hash_map_entry_create_deleted):
(pas_ptr_hash_map_entry_is_empty_or_deleted):
(pas_ptr_hash_map_entry_is_empty):
(pas_ptr_hash_map_entry_is_deleted):
(pas_ptr_hash_map_entry_get_key):
(pas_ptr_hash_map_key_get_hash):
(pas_ptr_hash_map_key_is_equal):
* libpas/src/libpas/pas_ptr_hash_set.h: Added.
(pas_ptr_hash_set_entry_create_empty):
(pas_ptr_hash_set_entry_create_deleted):
(pas_ptr_hash_set_entry_is_empty_or_deleted):
(pas_ptr_hash_set_entry_is_empty):
(pas_ptr_hash_set_entry_is_deleted):
(pas_ptr_hash_set_entry_get_key):
(pas_ptr_hash_set_key_get_hash):
(pas_ptr_hash_set_key_is_equal):
* libpas/src/libpas/pas_ptr_min_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_ptr_min_heap_compare):
(pas_ptr_min_heap_get_index):
(pas_ptr_min_heap_set_index):
* libpas/src/libpas/pas_ptr_worklist.c: Added.
(pas_ptr_worklist_construct):
(pas_ptr_worklist_destruct):
(pas_ptr_worklist_push):
(pas_ptr_worklist_pop):
* libpas/src/libpas/pas_ptr_worklist.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_race_test_hooks.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_race_test_hooks.h: Added.
(pas_race_test_hook_kind_get_string):
(pas_race_test_hook):
(pas_race_test_will_lock):
(pas_race_test_did_lock):
(pas_race_test_did_try_lock):
(pas_race_test_will_unlock):
* libpas/src/libpas/pas_random.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_random.h: Added.
(pas_get_random):
* libpas/src/libpas/pas_range.h: Added.
(pas_range_create):
(pas_range_create_empty):
(pas_range_create_forgiving):
(pas_range_is_empty):
(pas_range_size):
(pas_range_contains):
(pas_range_subsumes):
(pas_range_overlaps):
(pas_range_create_intersection):
(pas_range_compare):
* libpas/src/libpas/pas_range16.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_range16_create):
* libpas/src/libpas/pas_range_begin_min_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_range_begin_min_heap_compare):
(pas_range_begin_min_heap_get_index):
(pas_range_begin_min_heap_set_index):
* libpas/src/libpas/pas_range_locked_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_range_locked_mode_get_string):
* libpas/src/libpas/pas_range_min_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_range_compare_begin):
(pas_range_get_index):
(pas_range_set_index):
* libpas/src/libpas/pas_reallocate_free_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_reallocate_heap_teleport_rule.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_red_black_tree.c: Added.
(tree_insert):
(left_rotate):
(right_rotate):
(remove_fixup):
(pas_red_black_tree_insert):
(pas_red_black_tree_remove):
(pas_red_black_tree_size):
* libpas/src/libpas/pas_red_black_tree.h: Added.
(pas_red_black_tree_validate_enumerable):
(pas_red_black_tree_get_root):
(pas_red_black_tree_node_get_left):
(pas_red_black_tree_node_get_right):
(pas_red_black_tree_node_get_parent):
(pas_red_black_tree_node_get_color):
(pas_red_black_tree_set_root):
(pas_red_black_tree_node_set_left):
(pas_red_black_tree_node_set_right):
(pas_red_black_tree_node_set_parent):
(pas_red_black_tree_node_set_color):
(pas_red_black_tree_node_minimum):
(pas_red_black_tree_node_maximum):
(pas_red_black_tree_node_successor):
(pas_red_black_tree_node_predecessor):
(pas_red_black_tree_node_reset):
(pas_red_black_tree_construct):
(pas_red_black_tree_node_find_exact):
(pas_red_black_tree_find_exact):
(pas_red_black_tree_node_find_least_greater_than_or_equal):
(pas_red_black_tree_find_least_greater_than_or_equal):
(pas_red_black_tree_node_find_least_greater_than):
(pas_red_black_tree_find_least_greater_than):
(pas_red_black_tree_node_find_greatest_less_than_or_equal):
(pas_red_black_tree_find_greatest_less_than_or_equal):
(pas_red_black_tree_node_find_greatest_less_than):
(pas_red_black_tree_find_greatest_less_than):
(pas_red_black_tree_minimum):
(pas_red_black_tree_maximum):
(pas_red_black_tree_is_empty):
* libpas/src/libpas/pas_redundant_local_allocator_node.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_redundant_local_allocator_node_create):
* libpas/src/libpas/pas_redundant_local_allocator_node.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_reserved_memory_provider.c: Added.
(null_aligned_allocator):
(initialize_config):
(pas_reserved_memory_provider_construct):
(pas_reserved_memory_provider_try_allocate):
* libpas/src/libpas/pas_reserved_memory_provider.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_root.c: Added.
(count_static_heaps_callback):
(collect_static_heaps_callback):
(pas_root_construct):
(pas_root_create):
(default_reader):
(enumerator_reader):
(enumerator_recorder):
(pas_root_enumerate_for_libmalloc):
(pas_root_enumerate_for_libmalloc_with_root_after_zone):
(malloc_introspect_good_size):
(malloc_introspect_check):
(malloc_introspect_print):
(malloc_introspect_log):
(malloc_introspect_force_lock):
(malloc_introspect_force_unlock):
(malloc_introspect_statistics):
(pas_root_ensure_for_libmalloc_enumeration):
* libpas/src/libpas/pas_root.h: Added.
* libpas/src/libpas/pas_scavenger.c: Added.
(ensure_data_instance):
(get_time_in_milliseconds):
(timed_wait):
(scavenger_thread_main):
(pas_scavenger_did_create_eligible):
(pas_scavenger_notify_eligibility_if_needed):
(pas_scavenger_suspend):
(pas_scavenger_resume):
(pas_scavenger_clear_all_non_tlc_caches):
(pas_scavenger_clear_all_caches_except_remote_tlcs):
(pas_scavenger_clear_all_caches):
(pas_scavenger_decommit_free_memory):
(pas_scavenger_run_synchronously_now):
(pas_scavenger_perform_synchronous_operation):
* libpas/src/libpas/pas_scavenger.h: Added.
* libpas/src/libpas/pas_segmented_vector.h: Added.
* libpas/src/libpas/pas_segregated_biasing_directory.c: Added.
(pas_segregated_biasing_directory_create):
(take_first_eligible_loop_head_callback):
(take_first_eligible_create_new_view_callback):
(pas_segregated_biasing_directory_take_first_eligible):
(pas_segregated_biasing_directory_take_last_unused):
* libpas/src/libpas/pas_segregated_biasing_directory.h: Added.
(pas_segregated_biasing_directory_magazine_index):
* libpas/src/libpas/pas_segregated_biasing_view.c: Added.
(pas_segregated_biasing_view_create):
(pas_segregated_biasing_view_is_owned):
(pas_segregated_biasing_view_lock_ownership_lock):
(pas_segregated_biasing_view_should_table):
(pas_segregated_biasing_view_note_eligibility):
(pas_segregated_biasing_view_note_emptiness):
(pas_segregated_biasing_view_compute_summary):
(pas_segregated_biasing_view_is_eligible):
(pas_segregated_biasing_view_is_empty):
* libpas/src/libpas/pas_segregated_biasing_view.h: Added.
(pas_segregated_biasing_view_as_view):
(pas_segregated_biasing_view_as_ineligible_view):
(pas_segregated_biasing_view_as_view_non_null):
(pas_segregated_biasing_view_as_ineligible_view_non_null):
* libpas/src/libpas/pas_segregated_directory.c: Added.
(pas_segregated_directory_construct):
(pas_segregated_directory_get_data_slow):
(pas_segregated_directory_get_use_epoch):
(pas_segregated_directory_get_sharing_payload):
(pas_segregated_directory_minimize_first_eligible):
(pas_segregated_directory_update_first_eligible_after_search):
(pas_segregated_directory_view_did_become_eligible_at_index_without_biasing_update):
(pas_segregated_directory_view_did_become_eligible_at_index):
(pas_segregated_directory_view_did_become_eligible_without_biasing_update):
(pas_segregated_directory_view_did_become_eligible):
(maximize_last_empty):
(pas_segregated_directory_view_did_become_empty_at_index):
(pas_segregated_directory_view_did_become_empty):
(pas_segregated_directory_is_committed):
(pas_segregated_directory_num_committed_views):
(num_empty_views_should_consider_view_parallel):
(pas_segregated_directory_num_empty_views):
(pas_segregated_directory_update_last_empty_plus_one_after_search):
(pas_segregated_directory_append):
(pas_segregated_directory_compute_summary):
(pas_segregated_directory_num_empty_granules):
* libpas/src/libpas/pas_segregated_directory.h: Added.
(pas_segregated_directory_size):
(pas_segregated_directory_bits_get_by_mask):
(pas_segregated_directory_bits_set_by_mask):
(pas_segregated_directory_get_misc_bit):
(pas_segregated_directory_set_misc_bit):
(pas_segregated_directory_get_other_misc_bit):
(pas_segregated_directory_set_other_misc_bit):
(pas_segregated_directory_spoof_inline_segment):
(pas_segregated_directory_get_empty_bit):
(pas_segregated_directory_set_empty_bit):
(pas_segregated_directory_get_data):
(pas_segregated_directory_can_do_sharing):
(pas_segregated_directory_is_doing_sharing):
(pas_segregated_directory_data_try_get_sharing_payload):
(pas_segregated_directory_start_sharing_if_necessary):
(pas_segregated_directory_data_get_first_eligible_ptr):
(pas_segregated_directory_is_eligible):
(pas_segregated_directory_is_empty):
(pas_segregated_directory_is_tabled):
(pas_segregated_directory_get_first_eligible_impl):
(pas_segregated_directory_get_first_eligible):
(pas_segregated_directory_get_first_eligible_torn):
(pas_segregated_directory_watch_first_eligible):
(pas_segregated_directory_get_last_empty_plus_one_impl):
(pas_segregated_directory_get_last_empty_plus_one):
(pas_segregated_directory_get_last_empty_plus_one_value):
(pas_segregated_directory_watch_last_empty_plus_one):
(pas_segregated_directory_last_empty_plus_one):
(pas_segregated_directory_get):
* libpas/src/libpas/pas_segregated_directory_bit_reference.h: Added.
(pas_segregated_directory_bit_reference_create_null):
(pas_segregated_directory_bit_reference_create_inline):
(pas_segregated_directory_bit_reference_create_out_of_line):
(pas_segregated_directory_bit_reference_is_null):
(pas_segregated_directory_bit_reference_is_inline):
(pas_segregated_directory_bit_reference_is_out_of_line):
* libpas/src/libpas/pas_segregated_directory_first_eligible_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_segregated_directory_find_eligible_kind_get_string):
* libpas/src/libpas/pas_segregated_directory_inlines.h: Added.
(pas_segregated_directory_iterate_iterate_callback):
(pas_segregated_directory_iterate_forward_iterate_callback):
(pas_segregated_directory_iterate_forward):
(pas_segregated_directory_iterate_backward_iterate_callback):
(pas_segregated_directory_iterate_backward):
(pas_segregated_directory_iterate_forward_to_take_first_eligible):
(pas_segregated_directory_iterate_backward_to_take_last_empty):
* libpas/src/libpas/pas_segregated_directory_kind.h: Added.
(pas_segregated_directory_kind_get_string):
* libpas/src/libpas/pas_segregated_exclusive_view.c: Added.
(pas_segregated_exclusive_view_create):
(pas_segregated_exclusive_view_should_table):
(pas_segregated_exclusive_ish_view_note_eligibility_impl):
(pas_segregated_exclusive_view_note_eligibility):
(pas_segregated_exclusive_view_note_emptiness):
(pas_segregated_exclusive_ish_view_compute_summary_impl):
(pas_segregated_exclusive_view_compute_summary):
(pas_segregated_exclusive_view_install_full_use_counts):
(pas_segregated_exclusive_view_is_eligible):
(pas_segregated_exclusive_view_is_empty):
* libpas/src/libpas/pas_segregated_exclusive_view.h: Added.
(pas_segregated_exclusive_view_as_view):
(pas_segregated_exclusive_view_as_ineligible_view):
(pas_segregated_exclusive_view_as_view_non_null):
(pas_segregated_exclusive_view_as_ineligible_view_non_null):
* libpas/src/libpas/pas_segregated_exclusive_view_inlines.h: Added.
(pas_segregated_exclusive_ish_view_did_start_allocating):
* libpas/src/libpas/pas_segregated_exclusive_view_ownership_kind.h: Added.
(pas_segregated_exclusive_view_ownership_kind_is_owned):
(pas_segregated_exclusive_view_ownership_kind_is_biased):
(pas_segregated_exclusive_view_ownership_kind_with_owned):
(pas_segregated_exclusive_view_ownership_kind_with_biased):
(pas_segregated_exclusive_view_ownership_kind_get_string):
* libpas/src/libpas/pas_segregated_global_size_directory.c: Added.
(pas_segregated_global_size_directory_create):
(pas_segregated_global_size_directory_ensure_data):
(pas_segregated_global_size_directory_get_extended_data):
(pas_segregated_global_size_directory_create_tlc_allocator):
(pas_segregated_global_size_directory_enable_exclusive_views):
(pas_segregated_global_size_directory_explode):
(pas_segregated_global_size_directory_select_allocator_slow):
(take_first_eligible_direct_loop_head_callback):
(take_first_eligible_direct_create_new_view_callback):
(pas_segregated_global_size_directory_take_first_eligible_direct):
(pas_segregated_global_size_directory_take_first_eligible):
(take_last_empty_should_consider_view_parallel):
(take_last_empty_consider_view):
(pas_segregated_global_size_directory_take_last_empty):
(pas_segregated_global_size_directory_for_object):
(pas_segregated_global_size_directory_get_allocator_from_tlc):
(pas_segregated_global_size_directory_compute_summary_for_unowned_exclusive):
(for_each_live_object_callback):
(pas_segregated_global_size_directory_for_each_live_object):
(pas_segregated_global_size_directory_local_allocator_size):
(pas_segregated_global_size_directory_num_allocator_indices):
(pas_segregated_global_size_directory_dump_reference):
(pas_segregated_global_size_directory_dump_for_spectrum):
* libpas/src/libpas/pas_segregated_global_size_directory.h: Added.
(pas_segregated_global_size_directory_as_view):
(pas_segregated_global_size_directory_alignment):
(pas_segregated_global_size_directory_contention_did_trigger_explosion):
(pas_segregated_global_size_directory_set_contention_did_trigger_explosion):
(pas_segregated_global_size_directory_has_tlc_allocator):
(pas_segregated_global_size_directory_are_exclusive_views_enabled):
(pas_segregated_global_size_directory_bias_sharing_pool):
* libpas/src/libpas/pas_segregated_global_size_directory_inlines.h: Added.
(pas_segregated_global_size_directory_select_allocator):
(pas_segregated_global_size_directory_local_allocator_size_for_null_config):
(pas_segregated_global_size_directory_local_allocator_size_for_config):
(pas_segregated_global_size_directory_num_allocator_indices_for_allocator_size):
(pas_segregated_global_size_directory_num_allocator_indices_for_config):
* libpas/src/libpas/pas_segregated_heap.c: Added.
(min_object_size_for_heap_config):
(max_count_for_page_config):
(max_object_size_for_page_config):
(max_small_count_for_heap_config):
(max_segregated_count_for_heap_config):
(max_bitfit_count_for_heap_config):
(max_count_for_heap_config):
(max_bitfit_object_size_for_heap_config):
(max_object_size_for_heap_config):
(pas_segregated_heap_construct):
(pas_segregated_heap_get_bitfit):
(medium_directory_tuple_for_index_impl):
(medium_directory_tuple_for_index_with_lock):
(pas_segregated_heap_medium_directory_tuple_for_index):
(pas_segregated_heap_medium_allocator_index_for_index):
(pas_segregated_heap_medium_size_directory_for_index):
(compute_small_index_upper_bound):
(ensure_count_lookup):
(pas_segregated_heap_ensure_allocator_index):
(compute_ideal_object_size):
(check_medium_directories):
(check_part_of_all_heaps_callback):
(check_part_of_all_segregated_heaps_callback):
(ensure_count_lookup_if_necessary):
(pas_segregated_heap_ensure_size_directory_for_count):
(pas_segregated_heap_get_num_free_bytes):
(pas_segregated_heap_for_each_global_size_directory):
(for_each_committed_size_directory_callback):
(pas_segregated_heap_for_each_committed_view):
(for_each_view_index_directory_callback):
(pas_segregated_heap_for_each_view_index):
(for_each_live_object_object_callback):
(for_each_live_object_directory_callback):
(for_each_live_object_bitfit_callback):
(pas_segregated_heap_for_each_live_object):
(num_committed_views_directory_callback):
(pas_segregated_heap_num_committed_views):
(num_empty_views_directory_callback):
(pas_segregated_heap_num_empty_views):
(num_empty_granules_directory_callback):
(pas_segregated_heap_num_empty_granules):
(num_views_directory_callback):
(pas_segregated_heap_num_views):
(compute_summary_directory_callback):
(pas_segregated_heap_compute_summary):
* libpas/src/libpas/pas_segregated_heap.h: Added.
(pas_segregated_heap_index_for_primitive_count):
(pas_segregated_heap_primitive_count_for_index):
(pas_segregated_heap_index_for_count):
(pas_segregated_heap_count_for_index):
(pas_segregated_heap_allocator_index_for_index):
(pas_segregated_heap_allocator_index_for_count_not_primitive):
* libpas/src/libpas/pas_segregated_heap_inlines.h: Added.
(pas_segregated_heap_size_directory_for_index):
(pas_segregated_heap_size_directory_for_count):
* libpas/src/libpas/pas_segregated_heap_lookup_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_segregated_heap_page_sharing_mode.h: Added.
(pas_segregated_heap_page_sharing_mode_does_any_sharing):
(pas_segregated_heap_page_sharing_mode_does_virtual_sharing):
(pas_segregated_heap_page_sharing_mode_does_physical_sharing):
(pas_segregated_heap_page_sharing_mode_get_string):
* libpas/src/libpas/pas_segregated_page.c: Added.
(pas_segregated_page_switch_lock_slow):
(pas_segregated_page_switch_lock_and_rebias_to_magazine_while_ineligible_impl):
(pas_segregated_page_construct):
(pas_segregated_page_note_emptiness):
(commit_lock_for):
(pas_segregated_page_take_empty_granules):
(pas_segregated_page_take_physically):
(pas_segregated_page_commit_fully):
(verify_granules_live_object_callback):
(pas_segregated_page_verify_granules):
(pas_segregated_page_deallocation_did_fail):
(pas_segregated_page_get_num_empty_granules):
(pas_segregated_page_get_num_committed_granules):
(pas_segregated_page_get_config):
(pas_segregated_page_add_commit_range):
(pas_segregated_page_and_config_for_address_and_heap_config):
(pas_segregated_page_verify_num_non_empty_words):
* libpas/src/libpas/pas_segregated_page.h: Added.
(pas_segregated_page_header_size):
(pas_segregated_page_offset_from_page_boundary_to_first_object_for_hugging_mode):
(pas_segregated_page_offset_from_page_boundary_to_end_of_last_object_for_hugging_mode):
(pas_segregated_page_useful_object_payload_size_for_hugging_mode):
(pas_segregated_page_best_hugging_mode):
(pas_segregated_page_offset_from_page_boundary_to_first_object_exclusive):
(pas_segregated_page_offset_from_page_boundary_to_end_of_last_object_exclusive):
(pas_segregated_page_useful_object_payload_size):
(pas_segregated_page_number_of_objects):
(pas_segregated_page_bytes_dirtied_per_object):
(pas_segregated_page_get_granule_use_counts):
(pas_segregated_page_for_boundary):
(pas_segregated_page_for_boundary_or_null):
(pas_segregated_page_for_boundary_unchecked):
(pas_segregated_page_boundary):
(pas_segregated_page_boundary_or_null):
(pas_segregated_page_for_address_and_page_config):
(pas_segregated_page_is_allocated_with_page):
(pas_segregated_page_is_allocated):
(pas_segregated_page_for_address_and_heap_config):
* libpas/src/libpas/pas_segregated_page_and_config.h: Added.
(pas_segregated_page_and_config_create):
(pas_segregated_page_and_config_create_empty):
(pas_segregated_page_and_config_is_empty):
* libpas/src/libpas/pas_segregated_page_config.c: Added.
(pas_segregated_page_config_validate):
* libpas/src/libpas/pas_segregated_page_config.h: Added.
(pas_segregated_page_config_is_enabled):
(pas_segregated_page_config_min_align):
(pas_segregated_page_config_object_payload_end_offset_from_boundary):
(pas_segregated_page_config_num_alloc_words):
(pas_segregated_page_config_num_alloc_bytes):
(pas_segregated_page_config_get_kind):
(pas_segregated_page_config_kind_is_utility):
(pas_segregated_page_config_is_utility):
(pas_segregated_page_config_kind_heap_lock_hold_mode):
(pas_segregated_page_config_heap_lock_hold_mode):
* libpas/src/libpas/pas_segregated_page_config_inlines.h: Added.
* libpas/src/libpas/pas_segregated_page_config_kind.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_segregated_page_config_kind_for_each):
* libpas/src/libpas/pas_segregated_page_config_kind.def: Added.
* libpas/src/libpas/pas_segregated_page_config_kind.h: Added.
(pas_segregated_page_config_kind_get_string):
(pas_segregated_page_config_kind_get_config):
* libpas/src/libpas/pas_segregated_page_config_utils.h: Added.
(pas_segregated_page_config_verify_dealloc):
* libpas/src/libpas/pas_segregated_page_config_utils_inlines.h: Added.
* libpas/src/libpas/pas_segregated_page_config_variant.h: Added.
(pas_segregated_page_config_variant_get_string):
* libpas/src/libpas/pas_segregated_page_emptiness_kind.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_page_emptiness_kind_get_inverted):
(pas_page_emptiness_kind_get_string):
* libpas/src/libpas/pas_segregated_page_granule_use_count.h: Added.
(pas_segregated_page_granule_use_count_increment):
(pas_segregated_page_granule_for_each_use_in_range):
(pas_segregated_page_granule_increment_uses_for_range):
* libpas/src/libpas/pas_segregated_page_header_placement_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_segregated_page_header_placement_mode_get_string):
* libpas/src/libpas/pas_segregated_page_inlines.h: Added.
(pas_segregated_page_offset_from_page_boundary_to_first_object):
(pas_segregated_page_initialize_full_use_counts):
(pas_segregated_page_lock_with_unbias_not_utility):
(pas_segregated_page_lock_with_unbias):
(pas_segregated_page_lock):
(pas_segregated_page_unlock):
(pas_segregated_page_switch_lock_with_mode):
(pas_segregated_page_switch_lock):
(pas_segregated_page_switch_lock_and_rebias_to_magazine_while_ineligible):
(pas_segregated_page_deallocate_with_page):
(pas_segregated_page_deallocate):
(pas_segregated_page_qualifies_for_decommit):
(pas_segregated_page_get_directory_for_address_in_page):
(pas_segregated_page_get_directory_for_address_and_page_config):
(pas_segregated_page_get_object_size_for_address_in_page):
(pas_segregated_page_get_object_size_for_address_and_page_config):
* libpas/src/libpas/pas_segregated_partial_view.c: Added.
(pas_segregated_partial_view_create):
(pas_segregated_partial_view_note_eligibility):
(pas_segregated_partial_view_set_is_in_use_for_allocation):
(pas_segregated_partial_view_should_table):
(compute_summary):
(pas_segregated_partial_view_compute_summary):
(pas_segregated_partial_view_is_eligible):
* libpas/src/libpas/pas_segregated_partial_view.h: Added.
(pas_segregated_partial_view_as_view):
(pas_segregated_partial_view_as_view_non_null):
* libpas/src/libpas/pas_segregated_partial_view_inlines.h: Added.
(pas_segregated_partial_view_tell_shared_handle_for_word_general_case_source):
(pas_segregated_partial_view_tell_shared_handle_for_word_general_case_callback):
(pas_segregated_partial_view_tell_shared_handle_for_word_general_case):
(pas_segregated_partial_view_did_start_allocating):
* libpas/src/libpas/pas_segregated_shared_handle.c: Added.
(pas_segregated_shared_handle_create):
(pas_segregated_shared_handle_destroy):
(pas_segregated_shared_handle_note_emptiness):
* libpas/src/libpas/pas_segregated_shared_handle.h: Added.
(pas_segregated_shared_handle_num_views):
(pas_segregated_shared_handle_size):
(pas_segregated_shared_handle_as_view):
(pas_segregated_shared_handle_as_view_non_null):
* libpas/src/libpas/pas_segregated_shared_handle_inlines.h: Added.
(pas_segregated_shared_handle_partial_view_ptr_for_index):
(pas_segregated_shared_handle_partial_view_for_index):
(pas_segregated_shared_handle_partial_view_for_object):
* libpas/src/libpas/pas_segregated_shared_page_directory.c: Added.
(find_first_eligible_should_consider_view_parallel):
(find_first_eligible_consider_view):
(pas_segregated_shared_page_directory_find_first_eligible):
(take_last_empty_should_consider_view_parallel):
(switch_to_ownership):
(take_last_empty_consider_view):
(pas_segregated_shared_page_directory_take_last_empty):
(pas_segregated_shared_page_directory_dump_reference):
(pas_segregated_shared_page_directory_dump_for_spectrum):
* libpas/src/libpas/pas_segregated_shared_page_directory.h: Added.
* libpas/src/libpas/pas_segregated_shared_view.c: Added.
(pas_segregated_shared_view_create):
(pas_segregated_shared_view_commit_page):
(compute_summary_for_each_live_object_callback):
(compute_summary):
(pas_segregated_shared_view_compute_summary):
(pas_segregated_shared_view_is_empty):
* libpas/src/libpas/pas_segregated_shared_view.h: Added.
(pas_segregated_shared_view_as_view):
(pas_segregated_shared_view_as_view_non_null):
(pas_shared_view_computed_bump_result_create_empty):
(pas_segregated_shared_view_compute_initial_new_bump):
(pas_segregated_shared_view_can_bump):
(pas_segregated_shared_view_compute_new_bump):
(pas_segregated_shared_view_bump):
* libpas/src/libpas/pas_segregated_shared_view_inlines.h: Added.
(pas_segregated_shared_view_commit_page_if_necessary):
* libpas/src/libpas/pas_segregated_size_directory.c: Added.
(pas_segregated_size_directory_construct):
(for_each_live_object_object_callback):
(pas_segregated_size_directory_for_each_live_object):
(pas_segregated_size_directory_for_object):
* libpas/src/libpas/pas_segregated_size_directory.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_segregated_size_directory_inlines.h: Added.
(pas_segregated_size_directory_take_first_eligible_impl_should_consider_view_not_tabled_parallel):
(pas_segregated_size_directory_take_first_eligible_impl_should_consider_view_tabled_parallel):
(pas_segregated_size_directory_take_first_eligible_impl):
(pas_segregated_size_directory_get_global):
* libpas/src/libpas/pas_segregated_view.c: Added.
(pas_segregated_view_get_size_directory_slow):
(pas_segregated_view_get_global_size_directory_slow):
(pas_segregated_view_get_page_config_kind):
(pas_segregated_view_get_page_config):
(pas_segregated_view_get_index):
(pas_segregated_view_get_page_boundary):
(pas_segregated_view_get_page):
(pas_segregated_view_should_restart):
(pas_segregated_view_could_bump):
(pas_segregated_view_get_commit_lock):
(pas_segregated_view_get_ownership_lock):
(pas_segregated_view_is_owned):
(pas_segregated_view_should_table):
(pas_segregated_view_is_biased_exclusive):
(pas_segregated_view_lock_ownership_lock):
(pas_segregated_view_lock_ownership_lock_conditionally):
(pas_segregated_view_lock_ownership_lock_if_owned):
(pas_segregated_view_lock_ownership_lock_if_owned_conditionally):
(pas_segregated_view_unlock_ownership_lock):
(pas_segregated_view_unlock_ownership_lock_conditionally):
(pas_segregated_view_is_primordial_partial):
(pas_segregated_view_note_eligibility):
(pas_segregated_view_note_emptiness):
(for_each_live_object):
(pas_segregated_view_for_each_live_object):
(should_be_eligible):
(pas_segregated_view_should_be_eligible):
(pas_segregated_view_for_object):
(pas_segregated_view_compute_summary):
(pas_segregated_view_is_eligible):
(pas_segregated_view_is_eligible_or_biased):
(is_payload_empty_callback):
(pas_segregated_view_is_payload_empty):
(pas_segregated_view_is_empty):
(pas_segregated_view_is_empty_or_biased):
* libpas/src/libpas/pas_segregated_view.h: Added.
(pas_segregated_view_create):
(pas_segregated_view_create_non_null):
(pas_segregated_view_get_ptr):
(pas_segregated_view_get_kind):
(pas_segregated_view_is_null):
(pas_segregated_view_is_exclusive):
(pas_segregated_view_is_ineligible_exclusive):
(pas_segregated_view_is_some_exclusive):
(pas_segregated_view_is_exclusive_ish):
(pas_segregated_view_is_biasing):
(pas_segregated_view_is_ineligible_biasing):
(pas_segregated_view_is_some_biasing):
(pas_segregated_view_get_biasing):
(pas_segregated_view_is_eligible_kind):
(pas_segregated_view_is_ineligible_kind):
(pas_segregated_view_as_ineligible):
(pas_segregated_view_as_eligible):
(pas_segregated_view_is_shared):
(pas_segregated_view_get_shared):
(pas_segregated_view_is_shared_handle):
(pas_segregated_view_get_shared_handle):
(pas_segregated_view_is_partial):
(pas_segregated_view_get_partial):
(pas_segregated_view_is_global_size_directory):
(pas_segregated_view_get_size_directory):
(pas_segregated_view_get_global_size_directory):
* libpas/src/libpas/pas_segregated_view_allocator_inlines.h: Added.
(pas_segregated_view_will_start_allocating):
(pas_segregated_view_did_stop_allocating):
* libpas/src/libpas/pas_segregated_view_inlines.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_segregated_view_get_exclusive):
* libpas/src/libpas/pas_segregated_view_kind.h: Added.
(pas_segregated_view_kind_get_character_code):
(pas_segregated_view_kind_get_string):
(pas_segregated_view_kind_is_eligible):
(pas_segregated_view_kind_is_exclusive_ish):
(pas_segregated_view_kind_is_ineligible):
(pas_segregated_view_kind_can_become_empty):
* libpas/src/libpas/pas_shared_handle_or_page.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_wrap_page):
(pas_is_wrapped_shared_handle):
(pas_is_wrapped_page):
(pas_unwrap_page):
* libpas/src/libpas/pas_shared_handle_or_page_boundary.h: Added.
(pas_wrap_page_boundary):
(pas_is_wrapped_shared_handle):
(pas_is_wrapped_page_boundary):
(pas_unwrap_page_boundary):
* libpas/src/libpas/pas_shared_handle_or_page_boundary_inlines.h: Added.
(pas_wrap_shared_handle):
(pas_unwrap_shared_handle_no_liveness_checks):
(pas_unwrap_shared_handle):
(pas_shared_handle_or_page_boundary_get_page_boundary_no_liveness_checks):
(pas_shared_handle_or_page_boundary_get_page_boundary):
* libpas/src/libpas/pas_shared_handle_or_page_inlines.h: Added.
(pas_wrap_shared_handle):
(pas_unwrap_shared_handle_no_liveness_checks):
(pas_unwrap_shared_handle):
(pas_shared_handle_or_page_get_page_no_liveness_checks):
(pas_shared_handle_or_page_get_page):
* libpas/src/libpas/pas_shared_page_directory_by_size.c: Added.
(pas_shared_page_directory_by_size_get):
(pas_shared_page_directory_by_size_for_each):
(pas_shared_page_directory_by_size_for_each_remote):
* libpas/src/libpas/pas_shared_page_directory_by_size.h: Added.
* libpas/src/libpas/pas_simple_free_heap_declarations.def: Added.
* libpas/src/libpas/pas_simple_free_heap_definitions.def: Added.
* libpas/src/libpas/pas_simple_free_heap_helpers.c: Added.
(pas_simple_free_heap_helpers_try_allocate_with_manual_alignment):
(pas_simple_free_heap_helpers_deallocate):
* libpas/src/libpas/pas_simple_free_heap_helpers.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_simple_large_free_heap.c: Added.
(free_list_entry):
(pas_simple_large_free_heap_construct):
(dump_free_list):
(consider_expanding):
(append):
(merge):
(remove_entry):
(index_to_cursor):
(cursor_to_index):
(simple_find_first):
(simple_find_by_end):
(simple_read_cursor):
(simple_write_cursor):
(simple_merge):
(simple_remove):
(simple_append):
(simple_commit):
(simple_dump):
(simple_add_mapped_bytes):
(initialize_generic_heap_config):
(merge_physical):
(try_allocate_without_fixing):
(fix_free_list_if_necessary):
(pas_simple_large_free_heap_try_allocate):
(pas_simple_large_free_heap_deallocate):
(pas_simple_large_free_heap_for_each_free):
(pas_simple_large_free_heap_get_num_free_bytes):
(pas_simple_large_free_heap_dump_to_printf):
* libpas/src/libpas/pas_simple_large_free_heap.h: Added.
* libpas/src/libpas/pas_simple_type.h: Added.
(pas_simple_type_has_key):
(pas_simple_type_get_key_data):
(pas_simple_type_unwrap):
(pas_simple_type_size):
(pas_simple_type_alignment):
(pas_simple_type_key):
(pas_simple_type_create):
(pas_simple_type_create_with_key_data):
(pas_simple_type_as_heap_type_get_type_size):
(pas_simple_type_as_heap_type_get_type_alignment):
* libpas/src/libpas/pas_size_thunk.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_trivial_size_thunk):
* libpas/src/libpas/pas_skip_list.c: Added.
(pas_skip_list_construct):
(pas_skip_list_node_allocate_with_height):
(pas_skip_list_node_allocate):
(pas_skip_list_node_deallocate):
(pas_skip_list_remove):
(pas_skip_list_size):
(validate_other_node):
(pas_skip_list_validate):
* libpas/src/libpas/pas_skip_list.h: Added.
(pas_skip_list_node_prev):
(pas_skip_list_node_next):
(pas_skip_list_head):
(pas_skip_list_is_empty):
* libpas/src/libpas/pas_skip_list_inlines.h: Added.
(pas_skip_list_find_result_create_exact):
(pas_skip_list_find_result_create_inexact):
(pas_skip_list_level_get_direction):
(pas_skip_list_find_impl):
(pas_skip_list_find_ignore_head_attachment):
(pas_skip_list_find_ignore_pole_attachment):
(pas_skip_list_find):
(pas_skip_list_insert_after_note_head_attachment):
(pas_skip_list_insert_after_note_pole_attachment):
(pas_skip_list_insert):
* libpas/src/libpas/pas_slow_path_mode.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_slow_path_mode_prefix.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_small_large_map_entry.h: Added.
(pas_small_large_map_entry_create_empty):
(pas_small_large_map_entry_create_deleted):
(pas_small_large_map_entry_is_empty_or_deleted):
(pas_small_large_map_entry_is_empty):
(pas_small_large_map_entry_is_deleted):
(pas_small_large_map_entry_create):
(pas_small_large_map_entry_begin):
(pas_small_large_map_entry_get_key):
(pas_small_large_map_entry_end):
(pas_small_large_map_entry_heap):
(pas_small_large_map_entry_get_entry):
(pas_small_large_map_entry_can_create):
* libpas/src/libpas/pas_snprintf.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_status_reporter.c: Added.
(dump_ratio_initial):
(dump_occupancy_initial):
(dump_arrow):
(report_bitfit_directory_contents):
(pas_status_reporter_dump_bitfit_biasing_directory):
(pas_status_reporter_dump_bitfit_global_directory):
(report_segregated_directory_contents):
(pas_status_reporter_dump_segregated_biasing_directory):
(pas_status_reporter_dump_segregated_global_size_directory):
(pas_status_reporter_dump_segregated_shared_page_directory):
(pas_status_reporter_dump_large_heap):
(pas_status_reporter_dump_large_map):
(pas_status_reporter_dump_heap_table):
(pas_status_reporter_dump_immortal_heap):
(pas_status_reporter_dump_compact_large_utility_free_heap):
(pas_status_reporter_dump_large_utility_free_heap):
(pas_status_reporter_dump_compact_bootstrap_free_heap):
(pas_status_reporter_dump_bootstrap_free_heap):
(dump_segregated_heap_directory_callback):
(pas_status_reporter_dump_bitfit_heap):
(pas_status_reporter_dump_segregated_heap):
(pas_status_reporter_dump_heap):
(dump_all_heaps_heap_callback):
(pas_status_reporter_dump_all_heaps):
(dump_all_shared_page_directories_directory_callback):
(pas_status_reporter_dump_all_shared_page_directories):
(pas_status_reporter_dump_all_heaps_non_utility_summaries):
(pas_status_reporter_dump_utility_heap):
(add_to_size_histogram):
(dump_histogram):
(total_fragmentation_global_size_directory_callback):
(total_fragmentation_heap_callback):
(total_fragmentation_shared_page_directory_callback):
(pas_status_reporter_dump_total_fragmentation):
(tier_up_rate_global_size_directory_callback):
(tier_up_rate_heap_callback):
(dump_directory_tier_up_rate):
(pas_status_reporter_dump_tier_up_rates):
(allocator_state):
(pas_status_reporter_dump_baseline_allocators):
(pas_status_reporter_dump_thread_local_caches):
(pas_status_reporter_dump_configuration):
(pas_status_reporter_dump_everything):
(status_reporter_thread_main):
(start_reporter):
(pas_status_reporter_start_if_necessary):
* libpas/src/libpas/pas_status_reporter.h: Added.
* libpas/src/libpas/pas_stream.c: Copied from Source/WTF/wtf/FastTLS.h.
(pas_stream_vprintf):
(pas_stream_printf):
* libpas/src/libpas/pas_stream.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_stream_print_comma):
* libpas/src/libpas/pas_string_stream.c: Added.
(string_stream_vprintf):
(pas_string_stream_destruct):
(pas_string_stream_reset):
(pas_string_stream_vprintf):
* libpas/src/libpas/pas_string_stream.h: Added.
(pas_string_stream_get_string):
(pas_string_stream_get_string_length):
* libpas/src/libpas/pas_subpage_map.c: Added.
(pas_subpage_map_add):
(pas_subpage_map_get):
(pas_subpage_map_get_full_base):
* libpas/src/libpas/pas_subpage_map.h: Added.
(pas_subpage_map_hashtable_entry_create_empty):
(pas_subpage_map_hashtable_entry_create_deleted):
(pas_subpage_map_hashtable_entry_is_empty_or_deleted):
(pas_subpage_map_hashtable_entry_is_empty):
(pas_subpage_map_hashtable_entry_is_deleted):
(pas_subpage_map_hashtable_entry_get_key):
(pas_subpage_map_hashtable_key_get_hash):
(pas_subpage_map_hashtable_key_is_equal):
* libpas/src/libpas/pas_subpage_map_entry.c: Added.
(set_packed_value):
(pas_subpage_map_entry_create):
(prepare_indices):
(pas_subpage_map_assert_commit_state):
(set_bits):
(pas_subpage_map_entry_commit):
(pas_subpage_map_entry_decommit):
* libpas/src/libpas/pas_subpage_map_entry.h: Added.
(pas_subpage_map_entry_full_base):
(pas_subpage_map_entry_bits):
* libpas/src/libpas/pas_thread_local_cache.c: Added.
(pas_thread_local_cache_size_for_allocator_index_capacity):
(deallocate):
(destroy):
(destructor):
(allocate_cache):
(dump_thread_diagnostics):
(pas_thread_local_cache_create):
(pas_thread_local_cache_destroy):
(pas_thread_local_cache_get_slow):
(pas_thread_local_cache_get_local_allocator_slow):
(pas_thread_local_cache_get_local_allocator_if_can_set_cache_slow):
(pas_thread_local_cache_stop_local_allocators):
(stop_local_allocators_if_necessary_set_bit_source):
(stop_local_allocators_if_necessary_set_bit_callback):
(pas_thread_local_cache_stop_local_allocators_if_necessary):
(process_deallocation_log_with_config):
(flush_deallocation_log):
(pas_thread_local_cache_flush_deallocation_log):
(suspend):
(resume):
(pas_thread_local_cache_for_all):
(pas_thread_local_cache_append_deallocation_slow):
(pas_thread_local_cache_shrink):
* libpas/src/libpas/pas_thread_local_cache.h: Added.
(pas_thread_local_cache_try_get):
(pas_thread_local_cache_is_guaranteed_to_destruct):
(pas_thread_local_cache_can_set):
(pas_thread_local_cache_set_impl):
(pas_thread_local_cache_get_already_initialized):
(pas_thread_local_cache_get_with_heap_lock_hold_mode):
(pas_thread_local_cache_get):
(pas_thread_local_cache_get_holding_heap_lock):
(pas_thread_local_cache_get_local_allocator_impl):
(pas_thread_local_cache_get_local_allocator):
(pas_thread_local_cache_try_get_local_allocator):
(pas_thread_local_cache_get_local_allocator_if_can_set_cache):
(pas_thread_local_cache_encode_object):
(pas_thread_local_cache_append_deallocation):
(pas_thread_local_cache_append_deallocation_with_size):
* libpas/src/libpas/pas_thread_local_cache_layout.c: Added.
(pas_thread_local_cache_layout_add_node):
(pas_thread_local_cache_layout_add):
(pas_thread_local_cache_layout_duplicate):
* libpas/src/libpas/pas_thread_local_cache_layout.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_thread_local_cache_layout_node.c: Added.
(pas_thread_local_cache_layout_node_get_directory):
(allocator_index_ptr):
(pas_thread_local_cache_layout_node_get_allocator_index):
(pas_thread_local_cache_layout_node_set_allocator_index):
(next_ptr):
(pas_thread_local_cache_layout_node_get_next):
(pas_thread_local_cache_layout_node_set_next):
* libpas/src/libpas/pas_thread_local_cache_layout_node.h: Added.
(pas_wrap_segregated_global_size_directory):
(pas_wrap_redundant_local_allocator_node):
(pas_is_wrapped_segregated_global_size_directory):
(pas_is_wrapped_redundant_local_allocator_node):
(pas_unwrap_segregated_global_size_directory):
(pas_unwrap_redundant_local_allocator_node):
* libpas/src/libpas/pas_thread_local_cache_node.c: Added.
(pas_thread_local_cache_node_allocate):
(pas_thread_local_cache_node_deallocate):
* libpas/src/libpas/pas_thread_local_cache_node.h: Added.
* libpas/src/libpas/pas_tiny_large_map_entry.h: Added.
(pas_tiny_large_map_entry_base):
(pas_tiny_large_map_entry_create):
(pas_tiny_large_map_entry_begin):
(pas_tiny_large_map_entry_end):
(pas_tiny_large_map_entry_heap):
(pas_tiny_large_map_entry_get_entry):
(pas_tiny_large_map_entry_can_create):
(pas_tiny_large_map_entry_create_empty):
(pas_tiny_large_map_entry_create_deleted):
(pas_tiny_large_map_entry_is_empty_or_deleted):
(pas_tiny_large_map_entry_is_empty):
(pas_tiny_large_map_entry_is_deleted):
(pas_tiny_large_map_entry_get_key):
(pas_tiny_large_map_key_get_hash):
(pas_tiny_large_map_key_is_equal):
* libpas/src/libpas/pas_tree_direction.h: Added.
(pas_tree_direction_get_string):
(pas_tree_direction_invert):
(pas_tree_direction_invert_comparison_result_if_right):
* libpas/src/libpas/pas_tri_state.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_tri_state_get_string):
(pas_tri_state_equals_boolean):
* libpas/src/libpas/pas_try_allocate.h: Added.
(pas_try_allocate_impl_size_thunk):
(pas_try_allocate_impl):
* libpas/src/libpas/pas_try_allocate_array.h: Added.
(pas_try_allocate_array_impl):
* libpas/src/libpas/pas_try_allocate_common.h: Added.
(pas_try_allocate_common_can_go_fast):
(pas_try_allocate_common_impl_fast):
(pas_try_allocate_common_impl_slow):
(pas_try_allocate_common_impl):
* libpas/src/libpas/pas_try_allocate_intrinsic_primitive.h: Added.
(pas_try_allocate_intrinsic_primitive_impl_medium_slow_case):
(pas_try_allocate_intrinsic_primitive_impl):
* libpas/src/libpas/pas_try_allocate_primitive.h: Added.
(pas_try_allocate_primitive_impl):
* libpas/src/libpas/pas_try_reallocate.h: Added.
(pas_try_allocate_for_reallocate_and_copy):
(pas_try_reallocate_table_segregated_case):
(pas_try_reallocate_table_bitfit_case):
(pas_try_reallocate):
(pas_try_reallocate_intrinsic_primitive_allocate_callback):
(pas_try_reallocate_intrinsic_primitive):
(pas_try_reallocate_single_allocate_callback):
(pas_try_reallocate_single):
(pas_try_reallocate_array_allocate_callback):
(pas_try_reallocate_array):
(pas_try_reallocate_primitive_allocate_callback):
(pas_try_reallocate_primitive):
* libpas/src/libpas/pas_try_shrink.h: Added.
(pas_try_shrink):
* libpas/src/libpas/pas_typed_allocation_result.h: Added.
(pas_typed_allocation_result_create_empty):
(pas_typed_allocation_result_create):
(pas_typed_allocation_result_as_intrinsic_allocation_result):
(pas_typed_allocation_result_create_with_intrinsic_allocation_result):
(pas_typed_allocation_result_zero):
(pas_typed_allocation_result_set_errno):
(pas_typed_allocation_result_crash_on_error):
* libpas/src/libpas/pas_utility_heap.c: Added.
(pas_utility_heap_allocate_with_alignment):
(pas_utility_heap_try_allocate):
(pas_utility_heap_allocate):
(pas_utility_heap_deallocate):
(pas_utility_heap_get_num_free_bytes):
(for_each_live_object_small_object_callback):
(pas_utility_heap_for_each_live_object):
(pas_utility_heap_for_all_allocators):
* libpas/src/libpas/pas_utility_heap.h: Added.
(pas_utility_heap_allocate_with_asserted_kind):
(pas_utility_heap_deallocate_with_ignored_size_and_asserted_kind):
* libpas/src/libpas/pas_utility_heap_config.c: Added.
(pas_utility_heap_allocate_page):
(pas_utility_heap_config_for_each_shared_page_directory):
* libpas/src/libpas/pas_utility_heap_config.h: Added.
(pas_utility_heap_page_header_for_boundary):
(pas_utility_heap_boundary_for_page_header):
(pas_utility_heap_shared_page_directory_selector):
(pas_utility_heap_create_page_header):
(pas_utility_heap_destroy_page_header):
(pas_heap_config_is_utility):
(pas_heap_config_heap_lock_hold_mode):
* libpas/src/libpas/pas_utility_heap_support.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_utils.c: Added.
(pas_panic):
(pas_assertion_failed):
(pas_deallocation_did_fail):
(pas_set_deallocation_did_fail_callback):
(pas_reallocation_did_fail):
(pas_set_reallocation_did_fail_callback):
* libpas/src/libpas/pas_utils.h: Added.
(pas_is_power_of_2):
(pas_round_down_to_power_of_2):
(pas_round_down):
(pas_round_up_to_power_of_2):
(pas_round_up):
(pas_modulo_power_of_2):
(pas_is_aligned):
(pas_reverse):
(pas_reverse64):
(pas_make_mask64):
(pas_compare_and_swap_uintptr_weak):
(pas_compare_and_swap_uintptr_strong):
(pas_compare_and_swap_bool_weak):
(pas_compare_and_swap_bool_strong):
(pas_compare_and_swap_uint16_weak):
(pas_compare_and_swap_uint32_weak):
(pas_compare_and_swap_uint32_strong):
(pas_compare_and_swap_uint64_weak):
(pas_compare_and_swap_uint64_strong):
(pas_compare_and_swap_ptr_weak):
(pas_compare_and_swap_ptr_strong):
(pas_fence_after_load):
(pas_store_store_fence):
(pas_opaque):
(pas_pair_create):
(pas_compare_and_swap_pair_weak):
(pas_compare_and_swap_pair_strong):
(pas_atomic_load_pair):
(pas_atomic_store_pair):
(pas_hash32):
(pas_hash64):
(pas_hash_intptr):
(pas_hash_ptr):
(pas_log2):
(pas_log2_rounded_up):
(pas_log2_rounded_up_safe):
(pas_non_empty_ranges_overlap):
(pas_ranges_overlap):
(pas_xorshift32):
(pas_large_object_hash):
(pas_is_divisible_by):
* libpas/src/libpas/pas_utils_prefix.h: Added.
(__pas_round_up_to_power_of_2):
(__pas_compiler_fence):
(__pas_fence):
(__pas_depend_impl):
(__pas_depend):
(__pas_depend_cpu_only):
(__pas_memcpy):
* libpas/src/libpas/pas_versioned_field.c: Added.
(pas_versioned_field_minimize):
(pas_versioned_field_maximize):
(pas_versioned_field_minimize_watched):
(pas_versioned_field_maximize_watched):
* libpas/src/libpas/pas_versioned_field.h: Added.
(pas_versioned_field_construct):
(pas_versioned_field_create):
(pas_versioned_field_create_with_invalid_version):
(pas_versioned_field_create_empty):
(pas_versioned_field_with_version):
(pas_versioned_field_with_value):
(pas_versioned_field_as_pair):
(pas_versioned_field_from_pair):
(pas_versioned_field_equals):
(pas_versioned_field_read_torn):
(pas_versioned_field_read):
(pas_versioned_field_weak_cas):
(pas_versioned_field_strong_cas):
(pas_versioned_field_read_to_watch):
(pas_versioned_field_try_write):
(pas_versioned_field_try_write_watched):
(pas_versioned_field_write):
* libpas/src/libpas/pas_virtual_range.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/pas_virtual_range.h: Added.
(pas_virtual_range_create):
(pas_virtual_range_create_empty):
(pas_virtual_range_get_range):
(pas_virtual_range_is_empty):
(pas_virtual_range_size):
(pas_virtual_range_overlaps):
* libpas/src/libpas/pas_virtual_range_min_heap.h: Copied from Source/WTF/wtf/FastTLS.h.
(pas_virtual_range_compare_begin):
(pas_virtual_range_get_index):
(pas_virtual_range_set_index):
* libpas/src/libpas/pas_zero_mode.h: Added.
(pas_zero_mode_get_string):
(pas_zero_mode_validate):
(pas_zero_mode_merge):
* libpas/src/libpas/thingy_heap.c: Added.
(thingy_try_allocate_primitive):
(thingy_try_allocate_primitive_zeroed):
(thingy_try_allocate_primitive_with_alignment):
(thingy_try_reallocate_primitive):
(thingy_try_allocate):
(thingy_try_allocate_zeroed):
(thingy_try_allocate_array):
(thingy_try_allocate_zeroed_array):
(thingy_get_allocation_size):
(thingy_try_reallocate_array):
(thingy_deallocate):
(thingy_heap_ref_get_heap):
(thingy_utility_heap_allocate):
* libpas/src/libpas/thingy_heap.h: Added.
* libpas/src/libpas/thingy_heap_config.c: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/libpas/thingy_heap_config.h: Added.
* libpas/src/libpas/thingy_heap_prefix.h: Added.
* libpas/src/mbmalloc/mbmalloc_bmalloc.c: Copied from Source/WTF/wtf/FastTLS.h.
(mbmalloc):
(mbmemalign):
(mbrealloc):
(mbfree):
(mbscavenge):
* libpas/src/mbmalloc/mbmalloc_hotbit.c: Copied from Source/WTF/wtf/FastTLS.h.
(mbmalloc):
(mbmemalign):
(mbrealloc):
(mbfree):
(mbscavenge):
* libpas/src/mbmalloc/mbmalloc_iso_common_primitive.c: Added.
(install_verifier_if_necessary):
(mbmalloc):
(mbmemalign):
(mbrealloc):
(mbfree):
(dump_summary):
(mbscavenge):
* libpas/src/mbmalloc/mbmalloc_lineref.c: Copied from Source/WTF/wtf/FastTLS.h.
(mbmalloc):
(mbmemalign):
(mbrealloc):
(mbfree):
(mbscavenge):
* libpas/src/test/BitfieldVectorTests.cpp: Added.
(std::testBitfieldNumWords):
(std::testBitfieldNumFields):
(std::testBitfieldWordIndex):
(std::testBitfieldFieldIndex):
(std::testBitfieldFieldShift):
(std::BitfieldVector::BitfieldVector):
(std::BitfieldVector::get const):
(std::BitfieldVector::set):
(std::testBitfieldVectorForward):
(std::testBitfieldVectorBackward):
(addBitfieldVectorTests):
* libpas/src/test/BitvectorTests.cpp: Added.
(std::testFindFirst):
(addBitvectorTests):
* libpas/src/test/CartesianTreeTests.cpp: Added.
(std::TestNode::TestNode):
(std::TestNode::compare):
(std::TestNode::getXKey):
(std::TestNode::getYKey):
(std::Pair::Pair):
(std::Pair::operator== const):
(std::Pair::operator< const):
(std::findExact):
(std::remove):
(std::findLeastGreaterThanOrEqual):
(std::assertFoundAndRemove):
(std::assertEqual):
(std::assertSameValuesForKey):
(std::cartesianTreePrintWidth):
(std::printSpaces):
(std::dumpCartesianTree):
(std::dumpVector):
(std::testDriver):
(std::randomTestDriver):
(addCartesianTreeTests):
* libpas/src/test/CoalignTests.cpp: Added.
(std::testCoalignOneSided):
(std::testCoalignOneSidedError):
(std::testCoalign):
(addCoalignTests):
* libpas/src/test/ExtendedGCDTests.cpp: Copied from Source/WTF/wtf/FastTLS.h.
(std::testExtendedGCD):
(addExtendedGCDTests):
* libpas/src/test/HashtableTests.cpp: Added.
(std::hashtableForEachEntryCallback):
(std::hashtableForEachEntry):
(std::Key::Key):
(std::CollidingEntry_create_empty):
(std::CollidingEntry_create_deleted):
(std::CollidingEntry_is_empty_or_deleted):
(std::CollidingEntry_is_empty):
(std::CollidingEntry_is_deleted):
(std::CollidingEntry_get_key):
(std::CollidingKey_get_hash):
(std::CollidingKey_is_equal):
(std::testEmptyCollidingHashtable):
(std::testCollidingHashtableAddFindTakeImpl):
(std::testCollidingHashtableAddFindTake):
(std::testCollidingHashtableAddAddTakeSet):
(std::testCollidingHashtableAddAddAddTakeTakeSet):
(std::testCollidingHashtableAddAddAddTakeTakeAddSet):
(std::OutOfLineEntry_create_empty):
(std::OutOfLineEntry_create_deleted):
(std::OutOfLineEntry_is_empty_or_deleted):
(std::OutOfLineEntry_is_empty):
(std::OutOfLineEntry_is_deleted):
(std::OutOfLineEntry_get_key):
(std::OutOfLineKey_get_hash):
(std::OutOfLineKey_is_equal):
(std::testOutOfLineHashtable):
(addHashtableTests):
* libpas/src/test/HeapLocker.h: Copied from Source/WTF/wtf/FastTLS.h.
(HeapLocker::HeapLocker):
(HeapLocker::~HeapLocker):
* libpas/src/test/IsoDynamicPrimitiveHeapTests.cpp: Added.
(std::allocate42):
(std::allocate42WithAlignment):
(std::allocate42Zeroed):
(std::reallocate42):
(std::testManySizesAndKeys):
(std::testManySizesAndKeysInTandem):
(addIsoDynamicPrimitiveHeapTests):
* libpas/src/test/IsoHeapChaosTests.cpp: Added.
(std::flushDeallocationLogAndStopAllocators):
(std::verifyObjectSet):
(std::scavengerDidStart):
(std::scavengerWillShutDown):
(std::PageRange::PageRange):
(std::PageRange::operator< const):
(std::PageRange::end const):
(std::addPageRange):
(std::addPageRangeCallback):
(std::RecordedRange::RecordedRange):
(std::RecordedRange::operator< const):
(std::RecordedRange::end const):
(std::ReaderRange::ReaderRange):
(std::ReaderRange::operator< const):
(std::enumeratorReader):
(std::enumeratorRecorder):
(std::testAllocationChaos):
(std::addTheTests):
(std::addSpotTests):
(addIsoHeapChaosTests):
* libpas/src/test/IsoHeapPageSharingTests.cpp: Added.
(std::testTakePages):
(std::testTakePagesFromCorrectHeap):
(std::testLargeHeapTakesPagesFromCorrectSmallHeap):
(std::testLargeHeapTakesPagesFromCorrectSmallHeapAllocateAfterFree):
(std::testLargeHeapTakesPagesFromCorrectSmallHeapWithFancyOrder):
(std::testLargeHeapTakesPagesFromCorrectLargeHeap):
(std::testLargeHeapTakesPagesFromCorrectLargeHeapAllocateAfterFreeOnSmallHeap):
(std::testLargeHeapTakesPagesFromCorrectLargeHeapAllocateAfterFreeOnAnotherLargeHeap):
(std::testLargeHeapTakesPagesFromCorrectLargeHeapWithFancyOrder):
(std::testSmallHeapTakesPagesFromCorrectLargeHeap):
(std::testSmallHeapTakesPagesFromCorrectLargeHeapWithFancyOrder):
(std::testSmallHeapTakesPagesFromCorrectLargeHeapAllocateAfterFreeOnSmallHeap):
(std::testSmallHeapTakesPagesFromCorrectLargeHeapAllocateAfterFreeOnAnotherLargeHeap):
(std::thingyName):
(std::forEachThingyKind):
(std::setupThingy):
(std::cleanThingy):
(std::checkThingy):
(std::addObject):
(std::deleteThingy):
(std::allocationKindName):
(std::allocateThingiesImpl):
(std::allocateThingies):
(std::heapSummaryFor):
(std::assertOnlyDecommitted):
(std::testFullVdirToVdirObvious):
(std::testFullVdirToVdirObviousBackwards):
(std::testFullVdirToVdirOpportunistic):
(std::testFullVdirToVdirOpportunisticBackwards):
(std::testFullVdirToVdirNewAllocation):
(std::testFullVdirToVdirNewLateAllocation):
(std::testFullVdirToVdirNewDirAllocation):
(std::testFullVdirToVdirNewLateDirAllocation):
(std::testFullVdirToVdirNewLargeAllocation):
(std::testFullVdirToVdirNewLateLargeAllocation):
(std::testFullVdirToDir):
(std::testFullVdirToDirBackwardsTarget):
(std::testFullVdirToDirBackwardsSource):
(std::testFullVdirToDirNewAllocation):
(std::testFullVdirToDirNewLateAllocation):
(std::testFullVdirToDirNewDirAllocation):
(std::testFullVdirToDirNewLateDirAllocation):
(std::testFullVdirToDirNewLargeAllocation):
(std::testFullNotVdirButLargeToDirNewLargeAllocation):
(std::testFullVdirToDirNewLateLargeAllocation):
(std::testFullVdirToDirNewAllocationAlsoPhysical):
(std::testFullVdirToDirNewLateAllocationAlsoPhysical):
(std::testFullVdirToLarge):
(std::testFullVdirToLargeBackwardsTarget):
(std::testFullVdirToLargeBackwardsSource):
(std::testFullVdirToLargeNewAllocation):
(std::testFullVdirToLargeNewLateAllocation):
(std::testFullVdirToLargeNewDirAllocation):
(std::testFullVdirToLargeNewLateDirAllocation):
(std::testFullVdirToLargeNewLargeAllocation):
(std::testFullNotVdirButLargeToLargeNewLargeAllocation):
(std::testFullVdirToLargeNewLateLargeAllocation):
(std::testFullVdirToLargeNewAllocationAlsoPhysical):
(std::testFullVdirToLargeNewLateAllocationAlsoPhysical):
(std::testFullDirToVdir):
(std::testFullDirToVdirBackwards):
(std::testFullDirToVdirNewAllocation):
(std::testFullDirToVdirNewLateAllocation):
(std::testFullDirToDir):
(std::testFullDirToDirBackwards):
(std::testFullDirToDirWithThree):
(std::testFullDirToDirWithThreeBackwards):
(std::testFullDirToDirWithThreeNewAllocation):
(std::testFullDirToDirWithThreeNewLateAllocation):
(std::testFullDirToDirWithThreeNewVdirAllocation):
(std::testFullDirToDirWithThreeNewLateVdirAllocation):
(std::testFullDirToLarge):
(std::testFullDirToLargeNewAllocation):
(std::testFullDirToLargeNewLateAllocation):
(std::testFullDirToLargeNewVdirAllocation):
(std::testFullDirToLargeNewLateVdirAllocation):
(std::testFullLargeToVdirForwardMinEpoch):
(std::testFullNotLargeButDirToVdirCombinedUseEpoch):
(std::testFullLargeToVdirCombinedUseEpoch):
(std::testFullLargeToVdirBackwards):
(std::testFullLargeToVdirNewAllocation):
(std::testFullLargeToVdirNewLateAllocation):
(std::testFullLargeToVdirNewDirAllocationForwardMinEpoch):
(std::testFullLargeToVdirNewDirAllocationCombinedUseEpoch):
(std::testFullLargeToVdirNewLateDirAllocationForwardMinEpoch):
(std::testFullLargeToVdirNewLateDirAllocationCombinedUseEpoch):
(std::testFullLargeToDirForwardMinEpoch):
(std::testFullLargeToDirCombinedUseEpoch):
(std::testFullLargeToDirBackwardsSource):
(std::testFullLargeToDirBackwardsTarget):
(std::testFullLargeToDirBackwardsSourceAndTarget):
(std::testFullLargeToDirNewAllocation):
(std::testFullLargeToDirNewLateAllocation):
(std::testFullLargeToDirNewVdirAllocationForwardMinEpoch):
(std::testFullLargeToDirNewVdirAllocationCombinedUseEpoch):
(std::testFullLargeToDirNewLateVdirAllocationForwardMinEpoch):
(std::testFullLargeToDirNewLateVdirAllocationCombinedUseEpoch):
(std::testFullLargeToDirNewDirAllocationForwardMinEpoch):
(std::testFullLargeToDirNewDirAllocationCombinedUseEpoch):
(std::testFullLargeToDirNewLateDirAllocationForwardMinEpoch):
(std::testFullLargeToDirNewLateDirAllocationCombinedUseEpoch):
(std::testFullLargeToLargeForwardMinEpoch):
(std::testFullLargeToLargeCombinedUseEpoch):
(std::testFullLargeToLargeReverse):
(std::testFullLargeToLargeNewAllocation):
(std::testFullLargeToLargeNewLateAllocation):
(std::testFullLargeToLargeNewVdirAllocation):
(std::testFullLargeToLargeNewLateVdirAllocation):
(std::testFullLargeToLargeNewDirAllocation):
(std::testFullLargeToLargeNewLateDirAllocation):
(std::testNewEligibleHasOlderEpoch):
(std::scavengerCompletionCallback):
(std::testScavengerEventuallyReturnsMemory):
(std::testScavengerEventuallyReturnsMemoryEvenWithoutManualShrink):
(std::scavengerShutDownCallback):
(std::testScavengerShutsDownEventually):
(std::addAllTests):
(addIsoHeapPageSharingTests):
* libpas/src/test/IsoHeapPartialAndBaselineTests.cpp: Added.
(std::FreeOrder::setCount const):
(std::FreeOrder::didSetCount const):
(std::forEachSharedPageDirectoryCallbackAdaptor):
(std::forEachSharedPageDirectory):
(std::numSharedPages):
(std::numCommittedSharedPages):
(std::testSimplePartialAllocations):
(std::testFreeAroundPrimordialStop):
(std::testFreeInterleavedAroundPrimordialStop):
(std::PartialProgram::PartialProgram):
(std::testMultiplePartialsFromDifferentHeapsPerShared):
(std::addMultiplePartialsFromDifferentHeapsPerSharedTests):
(std::testMultiplePartialsFromDifferentThreadsPerShared):
(std::incrementalRandom):
(std::zeroRandom):
(std::testTwoBaselinesEvictions):
(std::addScavengerDependentTests):
(addIsoHeapPartialAndBaselineTests):
* libpas/src/test/IsoHeapReservedMemoryTests.cpp: Added.
(std::testSizeProgression):
(addIsoHeapReservedMemoryTests):
* libpas/src/test/IsoHeapTablingTests.cpp: Added.
(std::testTabling):
(std::testUntabling):
(std::addTablingTests):
(addIsoHeapTablingTests):
* libpas/src/test/JITHeapTests.cpp: Added.
(std::testAllocateShrinkAndAllocate):
(addJITHeapTests):
* libpas/src/test/LargeFreeHeapTests.cpp: Added.
(std::alignSimple):
(std::trappingAllocator):
(std::trappingDeallocator):
(std::failingAllocator):
(std::Action::allocate):
(std::Action::deallocate):
(std::Free::Free):
(std::Free::operator== const):
(std::Free::operator< const):
(std::Allocation::Allocation):
(std::function<pas_aligned_allocation_result):
(std::function<void):
(std::operator<<):
(std::allocatorAdapter):
(std::deallocatorAdapter):
(std::iterateAdapter):
(std::iterateHeap):
(std::testLargeFreeHeapImpl):
(std::testSimpleLargeFreeHeap):
(std::testFastLargeFreeHeap):
(std::testBootstrapHeap):
(std::freeListSize):
(addLargeFreeHeapTests):
* libpas/src/test/LargeSharingPoolDump.cpp: Copied from Source/WTF/wtf/FastTLS.h.
(std::forEachAdapter):
(forEachLargeSharingPoolNode):
(dumpLargeSharingPool):
* libpas/src/test/LargeSharingPoolDump.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/src/test/LargeSharingPoolTests.cpp: Added.
(std::Range::Range):
(std::assertState):
(std::testBadCoalesceEpochUpdate):
(addLargeSharingPoolTests):
* libpas/src/test/LockFreeReadPtrPtrHashtableTests.cpp: Added.
(std::hashFunction):
(std::testChaos):
(addLockFreeReadPtrPtrHashtableTests):
* libpas/src/test/MinHeapTests.cpp: Added.
(std::Element::Element):
(std::asChar):
(std::compare):
(std::getIndex):
(std::setIndex):
(std::dumpHeap):
(std::testMinHeap):
(std::minHeapChaos):
(std::testMinHeapChaos):
(std::makeBootstrapAllocationConfig):
(std::testBootstrapMinHeapChaos):
(addMinHeapTests):
* libpas/src/test/RaceTests.cpp: Added.
(std::hookCallbackAdapter):
(std::willLockCallback):
(std::recordLockAcquisition):
(std::didLockCallback):
(std::didTryLockCallback):
(std::willUnlockCallback):
(std::isHoldingContendedLocks):
(std::InstallRaceHooks::InstallRaceHooks):
(std::testLocalAllocatorStopRace):
(std::testLocalAllocatorStopRaceAgainstScavenge):
(addRaceTests):
* libpas/src/test/RedBlackTreeTests.cpp: Added.
(std::TestNode::TestNode):
(std::TestNode::compare):
(std::TestNode::compareKey):
(std::Pair::Pair):
(std::Pair::operator== const):
(std::Pair::operator< const):
(std::findExact):
(std::remove):
(std::findLeastGreaterThanOrEqual):
(std::assertFoundAndRemove):
(std::assertEqual):
(std::assertSameValuesForKey):
(std::push):
(std::validateEnumerable):
(std::testDriver):
(std::randomTestDriver):
(addRedBlackTreeTests):
* libpas/src/test/SkipListTests.cpp: Added.
(std::TestNode::TestNode):
(std::TestNode::fromSkipListNode):
(std::TestNode::compareKey):
(std::TestNode::compareKeyLeastGreaterThanOrEqual):
(std::TestNode::compareKeyForInsert):
(std::TestNode::compare):
(std::Pair::Pair):
(std::Pair::operator== const):
(std::Pair::operator< const):
(std::findExact):
(std::remove):
(std::findLeastGreaterThanOrEqual):
(std::assertFoundAndRemove):
(std::assertEqual):
(std::assertSameValuesForKey):
(std::dumpSkipList):
(std::dumpVector):
(std::testDriver):
(std::randomTestDriver):
(addSkipListTests):
* libpas/src/test/SuspendScavenger.h: Copied from Source/WTF/wtf/FastTLS.h.
(SuspendScavenger::SuspendScavenger):
(SuspendScavenger::~SuspendScavenger):
* libpas/src/test/TSDTests.cpp: Added.
(std::destructor):
(std::testTSD):
(addTSDTests):
* libpas/src/test/TestHarness.cpp: Added.
(allocationConfigAllocate):
(allocationConfigDeallocate):
(TestScopeImpl::TestScopeImpl):
(RuntimeConfigTestScope::RuntimeConfigTestScope):
(ForceExclusives::ForceExclusives):
(ForceTLAs::ForceTLAs):
(ForceBitfit::ForceBitfit):
(DisableBitfit::DisableBitfit):
(ForcePartials::ForcePartials):
(ForceBaselines::ForceBaselines):
(RunScavengerFully::RunScavengerFully):
(RunScavengerOnNonRemoteCaches::RunScavengerOnNonRemoteCaches):
(SuspendScavengerScope::SuspendScavengerScope):
(VerifyGranules::VerifyGranules):
(InstallVerifier::InstallVerifier):
(DisableExplosion::DisableExplosion):
(ForceExplosion::ForceExplosion):
(ForceOneMagazine::ForceOneMagazine):
(EpochIsCounter::EpochIsCounter):
(BootJITHeap::BootJITHeap):
(testSucceeded):
(deterministicRandomNumber):
(addTest):
(TestScope::TestScope):
(TestScope::~TestScope):
(hasScope):
(dumpObjectSet):
(dumpObjectsInHeap):
(dumpObjectsInHeaps):
(forEachLiveObject):
(verifyMinimumObjectDistance):
(verifyExactObjectDistance):
(runOneTest):
(runForkedTest):
(runTests):
(runFilteredTests):
(main):
(scavenge):
(printStatusReport):
* libpas/src/test/TestHarness.h: Added.
(stringStreamConstruct):
(dumpToString):
(operator<<):
* libpas/src/test/ThingyAndUtilityHeapAllocationTests.cpp: Added.
(std::flushDeallocationLog):
(std::flushDeallocationLogAndStopAllocators):
(std::sizeClassFor):
(std::isLarge):
(std::forEachLiveUtilityObjectAdapter):
(std::forEachLiveUtilityObject):
(std::forEachCommittedViewAdapter):
(std::forEachCommittedView):
(std::numCommittedViews):
(std::numViews):
(std::verifyMinimumObjectDistance):
(std::verifyObjectSet):
(std::verifyUtilityObjectSet):
(std::verifyHeapEmpty):
(std::PrimitiveAllocator::PrimitiveAllocator):
(std::PrimitiveReallocAllocator::PrimitiveReallocAllocator):
(std::AlignedPrimitiveAllocator::AlignedPrimitiveAllocator):
(std::createIsolatedHeapRef):
(std::IsolatedHeapAllocator::IsolatedHeapAllocator):
(std::IsolatedHeapArrayAllocator::IsolatedHeapArrayAllocator):
(std::CounterScope::CounterScope):
(std::CounterScope::~CounterScope):
(std::testDeallocateNull):
(std::deallocationFailureCallback):
(std::DeallocationShouldFail::DeallocationShouldFail):
(std::DeallocationShouldFail::~DeallocationShouldFail):
(std::testDeallocateMalloc):
(std::testDeallocateStack):
(std::testSimpleAllocation):
(std::testSmallOrMediumAllocation):
(std::testSmallAllocation):
(std::testLargeAllocation):
(std::testAllocationWithInterleavedFragmentation):
(std::testFreeListRefillSpans):
(std::testInternalScavenge):
(std::testInternalScavengeFromCorrectDirectory):
(std::SizeClassProgram::SizeClassProgram):
(std::testSizeClassCreationImpl):
(std::testSizeClassCreation):
(std::testSpuriousEligibility):
(std::testBasicSizeClassNotSet):
(std::testSmallDoubleFree):
(std::testSmallFreeInner):
(std::testSmallFreeNextWithShrink):
(std::testSmallFreeNextWithoutShrink):
(std::testSmallFreeNextBeforeShrink):
(std::AllocationProgram::allocate):
(std::AllocationProgram::free):
(std::AllocationProgram::kind const):
(std::AllocationProgram::key const):
(std::AllocationProgram::isAllocate const):
(std::AllocationProgram::count const):
(std::AllocationProgram::alignment const):
(std::AllocationProgram::isFree const):
(std::IsolatedComplexAllocator::IsolatedComplexAllocator):
(std::IsolatedUnitComplexAllocator::IsolatedUnitComplexAllocator):
(std::checkObjectDistances):
(std::checkObjectBeginnings):
(std::addObjectAllocation):
(std::ExpectedBytes::exact):
(std::ExpectedBytes::upperBound):
(std::ExpectedBytes::check):
(std::testComplexLargeAllocationImpl):
(std::testComplexLargeAllocation):
(std::testAllocationCountProgression):
(std::testAllocationChaos):
(std::testUtilityAllocationChaos):
(std::testCombinedAllocationChaos):
(std::testLargeDoubleFree):
(std::testLargeOffsetFree):
(std::addDeallocationTests):
(std::testReallocatePrimitive):
(std::testReallocateArray):
(std::addSmallHeapTests):
(std::addLargeHeapTests):
(std::addMediumHeapTests):
(std::addLargerThanMediumHeapTests):
(std::addMargeBitfitTests):
(std::addLargerThanMargeBitfitTests):
(std::addCombinedHeapTests):
(std::addAllTestsImpl):
(std::addAllTests):
(addThingyAndUtilityHeapAllocationTests):
* libpas/src/test/UtilsTests.cpp: Added.
(std::testIsDivisibleBy3):
(addUtilsTests):
* libpas/src/toys/QuadraticSizeClass.cpp: Added.
(std::printSizeCell):
(std::printProgression):
(main):
* libpas/src/verifier/Verifier.cpp: Added.
(std::Locker::Locker):
(std::Locker::~Locker):
(std::Allocation::Allocation):
(std::allocationCallback):
(std::deallocationCallback):
(std::dumpStateForHeapKind):
(std::dumpStateHoldingLock):
(std::uninstall):
(pas_install_verifier):
(pas_dump_state):
* libpas/src/verifier/Verifier.h: Copied from Source/WTF/wtf/FastTLS.h.
* libpas/test-impl.sh: Added.
* libpas/test.sh: Added.
Source/JavaScriptCore:
Switch the ExecutableAllocator to the libpas jit_heap. The libpas jit_heap uses two size
categories of bitfit and a large heap, and is happy to do its approximate first-fit in any
ranges of memory you give it. Jit_heap never allocates metadata inside the memory it manages.
Allocations and deallocations take a constant-bounded amount of time except for unusual
situations, and are protected by fine-grained locking in most cases. Decommit follows libpas
scavenging policy. This alone is a speed-up on Speedometer (probably about 1% or more).
Also expose some libpas introspection via $vm.
* CMakeLists.txt:
* JavaScriptCore.xcodeproj/project.pbxproj:
* assembler/MacroAssemblerCodeRef.h:
* jit/ExecutableAllocator.cpp:
(JSC::ExecutableAllocator::underMemoryPressure):
(JSC::ExecutableAllocator::memoryPressureMultiplier):
(JSC::ExecutableAllocator::allocate):
(JSC::ExecutableAllocator::committedByteCount):
(JSC::ExecutableMemoryHandle::createImpl):
(JSC::ExecutableMemoryHandle::~ExecutableMemoryHandle):
(JSC::ExecutableMemoryHandle::sizeInBytes const):
(JSC::ExecutableMemoryHandle::shrink):
* jit/ExecutableAllocator.h:
* jit/ExecutableMemoryHandle.h: Added.
(JSC::ExecutableMemoryHandle::start const):
(JSC::ExecutableMemoryHandle::end const):
(JSC::ExecutableMemoryHandle::startAsInteger const):
(JSC::ExecutableMemoryHandle::endAsInteger const):
(JSC::ExecutableMemoryHandle::containsIntegerAddress const):
(JSC::ExecutableMemoryHandle::contains const):
(JSC::ExecutableMemoryHandle::key const):
(JSC::ExecutableMemoryHandle::dump const):
(JSC::ExecutableMemoryHandle::ExecutableMemoryHandle):
* tools/JSDollarVM.cpp:
(JSC::JSC_DEFINE_HOST_FUNCTION):
(JSC::JSDollarVM::finishCreation):
Source/WTF:
Introduce flags for using libpas as the ExecutableAllocator.
Explicitly ask for gigacage initialization, since libpas prefers it this way right now.
* wtf/FastTLS.h:
* wtf/PlatformUse.h:
* wtf/Threading.cpp:
(WTF::initialize):
Tools:
Disable the IsoHeap tests if we're using libpas. Libpas has its own tests.
* TestWebKitAPI/Tests/WTF/bmalloc/IsoHeap.cpp:
Canonical link: https://commits.webkit.org/239620@main
git-svn-id: https://svn.webkit.org/repository/webkit/trunk@279867 268f45cc-cd09-0410-ab3c-d52691b4dbfc
2021-07-13 07:39:34 +00:00
# if !defined(USE_LIBPAS_JIT_HEAP) && !USE(SYSTEM_MALLOC)
# include <bmalloc/BPlatform.h>
# if BENABLE(LIBPAS)
# if PLATFORM(MAC) && CPU(ARM64)
# define USE_LIBPAS_JIT_HEAP 1
# endif
# endif
# endif
