For arrays that hold ~150 million or more elements, we need to
allocate more than 4GB of ram. computeAllocBytes used to return a uint32_t so
we were truncating the real size and allocating a lot less memory than we
needed.
Reviewed By: @jdelong
Differential Revision: D1016732
Pretty much mechanical change, which will make it easier
to use the sign bit for RefCountStaticValue.
Reviewed By: @edwinsmith
Differential Revision: D1009397
Preparing for a bigger diff we are brining the APC files and objects to a common naming convention
Reviewed By: @jdelong
Differential Revision: D1007981
For non-packed HphpArrays, CopyReserve() was blindly copying m_pos from the
source array to the new array. This is incorrect because tombstones are not
copied over to the new array. This diff adds appropriate logic to recompute
m_pos for the new array.
Reviewed By: @jdelong
Differential Revision: D998944
{Packed,}ArrayInit is now the only way to create arrays while
assuming no write barriers are needed. To ensure it's safe, beef up
the assertions in their use. Fixed a few cases that had incorrect
size estimates.
Reviewed By: @markw65
Differential Revision: D980262
Every insert method for "Mixed" (set, setRef, etc) uses the
same code; factor it out. This approach to factoring avoids needing
to pass around strhash_t, which enables using a template;
string and int-specific key handling code is all inside the
overloaded insert() methods.
Reviewed By: @andralex
Differential Revision: D972465
This diff reworks the PHP extension compat layer to be more robust by
taking steps to reconcile the difference between mainstream PHP's way and
HHVM's way of representing program values and operating on program values.
This includes the shape of the object graph in the heap (what points to
what), how refcounting is done and how "reffiness" is handled, how copy-
on-write semantics are honored, and how values are stored into arrays.
One of the core ideas here is that HHVM supports "boxing" a program value
via the RefData type and that there's a reasonably clean mapping between
PHP's object graph and HHVM's object graph (when HHVM program values are
"boxed") that maps PHP "zval" objects onto HHVM RefData objects. For the
purposes of a compat layer for PHP extensions, it makes sense to use
RefData to represent "zvals". Of course, it is unreasonable to box every
program variable in HHVM since this would hurt performance. However, we
can make things work by boxing program values lazily as needed in the
right places to ensure that Zend PHP extensions only deal with boxed
values. Along the way I also fixed a bug where we were leaking all of
the arguments that were passed to Zend PHP extension functions.
The next issue involves "reffiness". At present, HHVM RefDatas are treated
as "reffy" when their refcount is 2 or greater, while Zend PHP zvals have a
separate flag (independent of the refcount) that indicates whether they
should be treated as "reffy". Zend PHP extensions are capable of setting
two or more program variable to point to the same zval where the zval's
"reffy" flag is set to false. In such cases, the program variables share
the zval using copy-on-write semantics, where the zval must be copied
before making a modification. To make this work, we need to change RefData
somehow so that we can keep track of whether copy-on-write is needed when
the refcount is 2 or greater. Adding "reffy" flag directly poses some
challenges in terms of performance, because it means that when a value's
refcount decreases from 2 to 1 we'll need to make sure the "reffy" flag
gets cleared. I was able to avoid this by adding two flags (m_cow and m_z)
and creating a scheme that maps the 3-tuple (m_count, m_cow, m_z) to the
2-tuple (realRefcount, reffy). Under this scheme decRef continues to work
as it does today, decrementing the m_count field and calling a helper
m_count reaches zero. Another neat thing about this scheme is that m_cow is
set to 1 iff copy-on-write would be required before modifying m_tv. (NOTE:
This diff does not implement copy-on-write yet for these cases, it will be
addressed in a later diff.)
Another issue is the relationship between strings/arrays and the
zvals/RefDatas that point to them. Under Zend PHP, it is assumed that a
zval exclusively owns the string or array and that nothing else points to
the string/array. A consequence of this is that Zend PHP extensions do not
perform any copy-on-write checks with the string or array's refcount before
modifying the string/array. HHVM on the other hand supports sharing strings
and arrays between multiple RefDatas and/or program variables. Thus, we
need some way to protect against Zend PHP extensions modifying a string or
array when its refcount is 2 or greater. This is achieved by putting checks
in the right places to lazily make a copy of the string or array (when
refcount >= 2) so that Zend PHP extensions only deal with strings and
arrays with a refcount of 1.
Finally, the Zend PHP APIs for adding values to an array are a little
different. Zend PHP extensions pass a zval to the API, and the appropriate
array slot is set to point at the zval. The API does not increment the
zval's refcount; it is the caller's responsibility to do this is needed.
New APIs were added to HphpArray to address this need.
With this in place, the next things to go after would be (1) Fixing up how
the Zend compat layer deals emulating Zend PHP string macros and
operations, this may require adding a new KindOfZStr type if StringData
can't be adapted without hurting HHVM performance; (2) Adding checks in the
right places to make a copy of a RefData if m_cow is 1, I've found most the
places where this is needed and it doesn't look like these checks will
noticably hurt performance; (3) improving support for resource types and
object types defined by Zend PHP extensions and interoperability with HHVM
extension classes, HHVM resources, and user-defined pure-PHP classes.
Reviewed By: @ptarjan
Differential Revision: D966781
They're only called from one place, and manually inlining them allows
a little bit of code sharing; maybe more over time.
Reviewed By: @jdelong
Differential Revision: D970506
They're used differently enough that this will aid streamlining each
one separately. Also, this way, key-type-specific logic in erase()
can be done earlier without extra branches.
Reviewed By: @jdelong
Differential Revision: D970271
Gets HphpArrays into a flat mode using MM().objMalloc()
instead of SmartAllocator. Various optimizations were needed to the
Make functions to get this to work out ok. Growth still creates a
non-flat HphpArray (leaving a PromotedPayload behind so we know how
big the original allocation was).
Reviewed By: @edwinsmith
Differential Revision: D969431
Various cleanups that I factored out of other work.
Eliminate ElmInd; it's usefulness is pretty low since we rely on
int32_t for the hashtable storage and ssize_t for locals and
and parameters that pass around array positions.
Renamed a few things for brevity & clarity.
Moved some inline functions to hphp-array-defs.h.
More use of auto, replace Elm* with auto& in several places.
Eliminate a bunch of unnecessary ssize_t() casts.
s/ElmIndEmpty/invalid_index in several places.
Reviewed By: bertrand
Differential Revision: D968744
This was a nice hack since lots of arrays are packed, however now that
enough are actually kPackedKind, this test doesn't appear to pay for
itself (not called as often as before, and nearly always false).
Reviewed By: @jdelong
Differential Revision: D963530
Get the data representation all at the end, and keep all
privates non-data members at the end too. Private ctors (eventually
should be removed and replaced by the Make functions) so we can change
its allocation behavior. The public members are still in a mostly
random order for now, and I left the bulk of the non-static private
member functions alone.
Reviewed By: @edwinsmith
Differential Revision: D960965
We only handle a few easy cases for these, but since these
operations renumber and compact elements on Mixed arrays,
there's no point in escalating from packed to mixed; we might
as well do memmove and leave the existing 0..N numbering.
Differential Revision: D945517
The AddLval methods were fairly cold, so reimplement them in
terms of lval(), but preserve the assert that ensures the keys
being added do not already exist. This reduces the number
of kind-specific methods we need to implement.
Added Array::setWithRef(key,value) to capture a common use
case for lvalAt(). Later this can implemented more efficiently
as a virtual method in ArrayData subclasses.
Differential Revision: D937227
Most strings used as keys are either small or not shared, where the
data pointers cannot be equal. Of the rest (shared), most could have
been small.
This diff copies small SharedVariant strings, so they don't refer
back to the SharedVariant, and don't need to be enlisted. Then,
this changes hitStringKey() to not check data pointers, making it
equivalent to StringData::same(). Lastly, hand-write the comparison
loop in StringData::same() so we compare words at a time, and the
loop can be inlined.
Differential Revision: D937233
Enable guarding tracelets on ArrayKind when we can generate
better code for a specific kind.
Added specialized helpers for CGetM, FPassM, and IssetM of
vector-shaped arrays. This eliminates an indirect call and allows the
entire array-access call stack to be inlined into the helper
Differential Revision: D873631
Brett Simmers