aboutsummaryrefslogtreecommitdiffstats
path: root/lib/Transforms/Vectorize/LoopVectorize.cpp
Commit message (Collapse)AuthorAgeFilesLines
* LoopVectorizer: Change variable name Stride to ConsecutiveStrideArnold Schwaighofer2013-04-241-6/+6
| | | | | | | | This makes it easier to read the code. No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180197 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Scalarize padded typesArnold Schwaighofer2013-04-241-1/+9
| | | | | | | | | | | | | | | | | | This patch disables memory-instruction vectorization for types that need padding bytes, e.g., x86_fp80 has 10 bytes store size with 6 bytes padding in darwin on x86_64. Because the load/store vectorization is performed by the bit casting to a packed vector, which has incompatible memory layout due to the lack of padding bytes, the present vectorizer produces inconsistent result for memory instructions of those types. This patch checks an equality of the AllocSize of a scalar type and allocated size for each vector element, to ensure that there is no padding bytes and the array can be read/written using vector operations. Patch by Daisuke Takahashi! Fixes PR15758. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180196 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Bail out if we don't have datalayout we need itArnold Schwaighofer2013-04-241-0/+5
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180195 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Fix 15830. When scalarizing and unrolling stores make sure ↵Nadav Rotem2013-04-231-4/+4
| | | | | | | | | | that the order in which the elements are scalarized is the same as the original order. This fixes a miscompilation in FreeBSD's regex library. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180121 91177308-0d34-0410-b5e6-96231b3b80d8
* Call the potentially costly isAnnotatedParallel() only once. Pekka Jaaskelainen2013-04-231-3/+5
| | | | | | | | Made the uniform write test's checks a bit stricter. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180119 91177308-0d34-0410-b5e6-96231b3b80d8
* Refuse to (even try to) vectorize loops which have uniform writes,Pekka Jaaskelainen2013-04-231-9/+9
| | | | | | | | | | | even if erroneously annotated with the parallel loop metadata. Fixes Bug 15794: "Loop Vectorizer: Crashes with the use of llvm.loop.parallel metadata" git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@180081 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Use matcher from PatternMatch.h for the min/max patternsArnold Schwaighofer2013-04-191-104/+102
| | | | | | | | | Also make some static function class functions to avoid having to mention the class namespace for enums all the time. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179886 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix a -Wdocumentation warningDmitri Gribenko2013-04-181-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179789 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Recognize min/max reductionsArnold Schwaighofer2013-04-181-34/+209
| | | | | | | | | | | | A min/max operation is represented by a select(cmp(lt/le/gt/ge, X, Y), X, Y) sequence in LLVM. If we see such a sequence we can treat it just as any other commutative binary instruction and reduce it. This appears to help bzip2 by about 1.5% on an imac12,2. radar://12960601 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179773 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Use a set to avoid longer cycles in the reduction chain too.Benjamin Kramer2013-04-181-8/+6
| | | | | | Fixes PR15748. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179757 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: integer division is not a reduction operationArnold Schwaighofer2013-04-121-2/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Don't classify idiv/udiv as a reduction operation. Integer division is lossy. For example : (1 / 2) * 4 != 4/2. Example: int a[] = { 2, 5, 2, 2} int x = 80; for() x /= a[i]; Scalar: x /= 2 // = 40 x /= 5 // = 8 x /= 2 // = 4 x /= 2 // = 2 Vectorized: <80, 1> / <2,5> //= <40,0> <40, 0> / <2,2> //= <20,0> 20*0 = 0 radar://13640654 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@179381 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Pass OperandValueKind information to the cost modelArnold Schwaighofer2013-04-041-2/+13
| | | | | | | | | | | | Pass down the fact that an operand is going to be a vector of constants. This should bring the performance of MultiSource/Benchmarks/PAQ8p/paq8p on x86 back. It had degraded to scalar performance due to my pervious shift cost change that made all shifts expensive on x86. radar://13576547 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@178809 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Invert case when we use a vector cmp value to query select costArnold Schwaighofer2013-03-141-1/+1
| | | | | | | We generate a select with a vectorized condition argument when the condition is NOT loop invariant. Not the other way around. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@177098 91177308-0d34-0410-b5e6-96231b3b80d8
* Remove a source of nondeterminism from the LoopVectorizer.Benjamin Kramer2013-03-091-1/+1
| | | | | | | This made us emit runtime checks in a random order. Hopefully bootstrap miscompares will go away now. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176775 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Ignore all dbg intrinisicArnold Schwaighofer2013-03-091-6/+6
| | | | | | Ignore all DbgIntriniscInfo instructions instead of just DbgValueInst. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176769 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Ignore dbg.value instructionsArnold Schwaighofer2013-03-091-2/+11
| | | | | | | | | We want vectorization to happen at -g. Ignore calls to the dbg.value intrinsic and don't transfer them to the vectorized code. radar://13378964 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176768 91177308-0d34-0410-b5e6-96231b3b80d8
* Insert the reduction start value into the first bypass block to preserve ↵Benjamin Kramer2013-03-081-1/+1
| | | | | | | | domination. Fixes PR15344. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176701 91177308-0d34-0410-b5e6-96231b3b80d8
* PR14448 - prevent the loop vectorizer from vectorizing the same loop twice.Nadav Rotem2013-03-021-0/+18
| | | | | | | | | | | | The LoopVectorizer often runs multiple times on the same function due to inlining. When this happens the loop vectorizer often vectorizes the same loops multiple times, increasing code size and adding unneeded branches. With this patch, the vectorizer during vectorization puts metadata on scalar loops and marks them as 'already vectorized' so that it knows to ignore them when it sees them a second time. PR14448. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176399 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Don't hang forever if a PHI only has skipped PHI uses.Benjamin Kramer2013-03-011-1/+8
| | | | | | Fixes PR15384. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176366 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Vectorize math builtin calls.Benjamin Kramer2013-02-271-50/+137
| | | | | | | | | | | This properly asks TargetLibraryInfo if a call is available and if it is, it can be translated into the corresponding LLVM builtin. We don't vectorize sqrt() yet because I'm not sure about the semantics for negative numbers. The other intrinsic should be exact equivalents to the libm functions. Differential Revision: http://llvm-reviews.chandlerc.com/D465 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@176188 91177308-0d34-0410-b5e6-96231b3b80d8
* Allow GlobalValues to vectorize with AliasAnalysisRenato Golin2013-02-211-35/+154
| | | | | | | | | | | | | | | | | | | | | Storing the load/store instructions with the values and inspect them using Alias Analysis to make sure they don't alias, since the GEP pointer operand doesn't take the offset into account. Trying hard to not add any extra cost to loads and stores that don't overlap on global values, AA is *only* calculated if all of the previous attempts failed. Using biggest vector register size as the stride for the vectorization access, as we're being conservative and the cost model (which calculates the real vectorization factor) is only run after the legalization phase. We might re-think this relationship in the future, but for now, I'd rather be safe than sorry. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175818 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Simplify code for clarity.Benjamin Kramer2013-02-131-10/+8
| | | | | | No functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175076 91177308-0d34-0410-b5e6-96231b3b80d8
* Metadata for annotating loops as parallel. The first consumer for this Pekka Jaaskelainen2013-02-131-0/+8
| | | | | | | | | | metadata is the loop vectorizer. See the documentation update for more info. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@175060 91177308-0d34-0410-b5e6-96231b3b80d8
* Typos.Jakob Stoklund Olesen2013-02-081-4/+4
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174723 91177308-0d34-0410-b5e6-96231b3b80d8
* ARM cost model: Address computation in vector mem ops not freeArnold Schwaighofer2013-02-081-8/+14
| | | | | | | | | | | | | | | Adds a function to target transform info to query for the cost of address computation. The cost model analysis pass now also queries this interface. The code in LoopVectorize adds the cost of address computation as part of the memory instruction cost calculation. Only there, we know whether the instruction will be scalarized or not. Increase the penality for inserting in to D registers on swift. This becomes necessary because we now always assume that address computation has a cost and three is a closer value to the architecture. radar://13097204 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174713 91177308-0d34-0410-b5e6-96231b3b80d8
* Test CommitMichael Kuperstein2013-02-081-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174709 91177308-0d34-0410-b5e6-96231b3b80d8
* fix 80-col violation and fix the docs.Nadav Rotem2013-02-071-3/+7
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174671 91177308-0d34-0410-b5e6-96231b3b80d8
* Loop Vectorizer: Refactor Memory Cost ComputationArnold Schwaighofer2013-02-071-180/+52
| | | | | | | | | | We don't want too many classes in a pass and the classes obscure the details. I was going a little overboard with object modeling here. Replace classes by generic code that handles both loads and stores. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174646 91177308-0d34-0410-b5e6-96231b3b80d8
* Loop Vectorizer: Refactor code to compute vectorized memory instruction costArnold Schwaighofer2013-02-051-79/+178
| | | | | | | Introduce a helper class that computes the cost of memory access instructions. No functionality change intended. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174422 91177308-0d34-0410-b5e6-96231b3b80d8
* Loop Vectorizer: Handle pointer stores/loads in getWidestType()Arnold Schwaighofer2013-02-051-9/+31
| | | | | | | | | | | | | | | | | In the loop vectorizer cost model, we used to ignore stores/loads of a pointer type when computing the widest type within a loop. This meant that if we had only stores/loads of pointers in a loop we would return a widest type of 8bits (instead of 32 or 64 bit) and therefore a vector factor that was too big. Now, if we see a consecutive store/load of pointers we use the size of a pointer (from data layout). This problem occured in SingleSource/Benchmarks/Shootout-C++/hash.cpp (reduced test case is the first test in vector_ptr_load_store.ll). radar://13139343 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@174377 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: convert TinyTripCountVectorThreshold constantPekka Jaaskelainen2013-01-291-1/+3
| | | | | | | | to a command line switch. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173837 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Clean up ValueMap a bit and avoid double lookups.Benjamin Kramer2013-01-291-10/+12
| | | | | | No intended functionality change. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173809 91177308-0d34-0410-b5e6-96231b3b80d8
* Vectorization Factor clarificationRenato Golin2013-01-281-17/+24
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173691 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Refactor the code that vectorizes loads/stores to remove ↵Nadav Rotem2013-01-251-145/+113
| | | | | | duplication. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173500 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Simplify code. No functionality change.Benjamin Kramer2013-01-251-26/+17
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173475 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Refactor more code to use the IRBuilder. Nadav Rotem2013-01-251-24/+25
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173471 91177308-0d34-0410-b5e6-96231b3b80d8
* Refactor some code to use the IRBuilder.Nadav Rotem2013-01-251-20/+21
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173467 91177308-0d34-0410-b5e6-96231b3b80d8
* Add support for reverse pointer induction variables. These are loops that ↵Nadav Rotem2013-01-231-7/+82
| | | | | | | | | | | | | | | contain pointers that count backwards. For example, this is the hot loop in BZIP: do { m = *--p; *p = ( ... ); } while (--n); git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173219 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix a comment. Induction vars dont need to start at zero.Nadav Rotem2013-01-211-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@173061 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorize: Fix a C++11 incompatibility.Benjamin Kramer2013-01-201-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172990 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix a build error.Nadav Rotem2013-01-201-2/+3
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172971 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Implement a new heuristics for selecting the unroll factor.Nadav Rotem2013-01-201-22/+65
| | | | | | | | | We ignore the cpu frontend and focus on pipeline utilization. We do this because we don't have a good way to estimate the loop body size at the IR level. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172964 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Emit memory checks into their own basic block.Benjamin Kramer2013-01-191-33/+53
| | | | | | | | | | | | | | This separates the check for "too few elements to run the vector loop" from the "memory overlap" check, giving a lot nicer code and allowing to skip the memory checks when we're not going to execute the vector code anyways. We still leave the decision of whether to emit the memory checks as branches or setccs, but it seems to be doing a good job. If ugly code pops up we may want to emit them as separate blocks too. Small speedup on MultiSource/Benchmarks/MallocBench/espresso. Most of this is legwork to allow multiple bypass blocks while updating PHIs, dominators and loop info. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172902 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer cost model. Honor the user command line flag that selects the ↵Nadav Rotem2013-01-151-1/+1
| | | | | | vectorization factor even if the target machine does not have any vector registers. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172544 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix PR14547. Handle induction variables of small sizes smaller than i32 (i8 ↵Nadav Rotem2013-01-131-0/+3
| | | | | | and i16). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172348 91177308-0d34-0410-b5e6-96231b3b80d8
* ARM Cost Model: We need to detect the max bitwidth of types in the loop in ↵Nadav Rotem2013-01-111-6/+12
| | | | | | | | | | | | | order to select the max vectorization factor. We don't have a detailed analysis on which values are vectorized and which stay scalars in the vectorized loop so we use another method. We look at reduction variables, loads and stores, which are the only ways to get information in and out of loop iterations. If the data types are extended and truncated then the cost model will catch the cost of the vector zext/sext/trunc operations. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172178 91177308-0d34-0410-b5e6-96231b3b80d8
* LoopVectorizer: Fix a bug in the vectorization of BinaryOperators. The ↵Nadav Rotem2013-01-101-4/+4
| | | | | | | | | | BinaryOperator can be folded to an Undef, and we don't want to set NSW flags to undef vals. PR14878 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172079 91177308-0d34-0410-b5e6-96231b3b80d8
* ARM Cost model: Use the size of vector registers and widest vectorizable ↵Nadav Rotem2013-01-091-3/+48
| | | | | | instruction to determine the max vectorization factor. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@172010 91177308-0d34-0410-b5e6-96231b3b80d8
* Cost Model: Move the 'max unroll factor' variable to the TTI and add initial ↵Nadav Rotem2013-01-091-3/+2
| | | | | | Cost Model support on ARM. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171928 91177308-0d34-0410-b5e6-96231b3b80d8
* Code cleanup: refactor the switch statements in the generation of reduction ↵Nadav Rotem2013-01-081-59/+28
| | | | | | variables into an IR builder call. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@171871 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.1 at 2025-11-12 08:12:37 +0000