aboutsummaryrefslogtreecommitdiffstats
path: root/lib/Transforms/Vectorize/SLPVectorizer.cpp
Commit message (Collapse)AuthorAgeFilesLines
* IRBuilder: Add RAII objects to reset insertion points or fast math flags.Benjamin Kramer2013-09-301-21/+1
| | | | | | | | Inspired by the object from the SLPVectorizer. This found a minor bug in the debug loc restoration in the vectorizer where the location of a following instruction was attached instead of the location from the original instruction. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191673 91177308-0d34-0410-b5e6-96231b3b80d8
* Even more spelling fixes for "instruction".Robert Wilhelm2013-09-281-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191611 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix SLPVectorizer using wrong address space for load/storeMatt Arsenault2013-09-271-4/+7
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191564 91177308-0d34-0410-b5e6-96231b3b80d8
* Transforms: Use getFirstNonPHI to set the insertion point for PHIsJustin Bogner2013-09-271-1/+1
| | | | | | | | | | We were previously using getFirstInsertionPt to insert PHI instructions when vectorizing, but getFirstInsertionPt also skips past landingpads, causing this to generate invalid IR. We can avoid this issue by using getFirstNonPHI instead. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191526 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorize: Put horizontal reductions feeding a store under separate flagArnold Schwaighofer2013-09-251-13/+18
| | | | | | | Put them under a separate flag for experimentation. They are more likely to interfere with loop vectorization which happens later in the pass pipeline. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191371 91177308-0d34-0410-b5e6-96231b3b80d8
* set the cost of tiny trees to INT_MAX in SLP vectorizer to disable ↵Yi Jiang2013-09-241-1/+1
| | | | | | vectorization on them git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191314 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix multiline comment warningArnold Schwaighofer2013-09-211-2/+2
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191135 91177308-0d34-0410-b5e6-96231b3b80d8
* Reapply "SLPVectorizer: Handle more horizontal reductions (disabled)""Arnold Schwaighofer2013-09-211-8/+364
| | | | | | | | | | | | | | Reapply r191108 with a fix for a memory corruption error I introduced. Of course, we can't reference the scalars that we replace by vectorizing and then call their eraseFromParent method. I only 'needed' the scalars to get the DebugLoc. Just store the DebugLoc before actually vectorizing instead. As a nice side effect, this also simplifies the interface between BoUpSLP and the HorizontalReduction class to returning a value pointer (the vectorized tree root). radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191123 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert "SLPVectorizer: Handle more horizontal reductions (disabled)"Arnold Schwaighofer2013-09-211-368/+8
| | | | | | | | | This reverts commit r191108. The horizontal.ll test case fails under libgmalloc. Thanks Shuxin for pointing this out to me. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191121 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Handle more horizontal reductions (disabled)Arnold Schwaighofer2013-09-201-8/+368
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Match reductions starting at binary operation feeding into a phi. The code handles trees like r += v1 + v2 + v3 ... and r += v1 r += v2 ... and r *= v1 + v2 + ... We currently only handle associative operations (add, fadd fast). The code can now also handle reductions feeding into stores. a[i] = v1 + v2 + v3 + ... The code is currently disabled behind the flag "-slp-vectorize-hor". The cost model for most architectures is not there yet. I found one opportunity of a horizontal reduction feeding a phi in TSVC (LoopRerolling-flt) and there are several opportunities where reductions feed into stores. radar://14607682 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@191108 91177308-0d34-0410-b5e6-96231b3b80d8
* Prevent LoopVectorizer and SLPVectorizer running if the target has no vector ↵Robert Lytton2013-09-181-0/+5
| | | | | | | | | | registers. XCore target: Add XCoreTargetTransformInfo This is where getNumberOfRegisters() resides, which in turn returns the number of vector registers (=0). git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190936 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert accidental commit I had to make to get the test case in PR17268 to ↵Craig Topper2013-09-181-1/+1
| | | | | | still work correctly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190917 91177308-0d34-0410-b5e6-96231b3b80d8
* Lift alignment restrictions for load/store folding on ↵Craig Topper2013-09-181-1/+1
| | | | | | VINSERTF128/VEXTRACTF128. Fixes PR17268. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190916 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Don't vectorize phi nodes that use invoke valuesArnold Schwaighofer2013-09-171-0/+12
| | | | | | | | | We can't insert an insertelement after an invoke. We would have to split a critical edge. So when we see a phi node that uses an invoke we just give up. radar://14990770 git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@190871 91177308-0d34-0410-b5e6-96231b3b80d8
* In this patch we are trying to do two things:Yi Jiang2013-09-031-11/+36
| | | | | | | | | 1) If the width of vectorization list candidate is bigger than vector reg width, we will break it down to fit the vector reg. 2) We do not vectorize the width which is not power of two. The performance result shows it will help some spec benchmarks. mesa improved 6.97% and ammp improved 1.54%. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189830 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix inserting instructions before last in bundle.Matt Arsenault2013-08-261-16/+20
| | | | | | | | | | | The builder inserts from before the insert point, not after, so this would insert before the last instruction in the bundle instead of after it. I'm not sure if this can actually be a problem with any of the current insertions. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189285 91177308-0d34-0410-b5e6-96231b3b80d8
* test commit. Remove blank lineYi Jiang2013-08-261-1/+0
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189265 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix unused variable in release buildMatt Arsenault2013-08-261-3/+2
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189264 91177308-0d34-0410-b5e6-96231b3b80d8
* Constify functionsMatt Arsenault2013-08-261-6/+8
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189234 91177308-0d34-0410-b5e6-96231b3b80d8
* Vectorize starting from insertelements building a vectorMatt Arsenault2013-08-261-3/+57
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189233 91177308-0d34-0410-b5e6-96231b3b80d8
* Check if in set on insertion instead of separatelyMatt Arsenault2013-08-241-7/+3
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189179 91177308-0d34-0410-b5e6-96231b3b80d8
* Teach the SLP vectorizer the correct way to check for consecutive accessChandler Carruth2013-08-221-49/+15
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | using GEPs. Previously, it used a number of different heuristics for analyzing the GEPs. Several of these were conservatively correct, but failed to fall back to SCEV even when SCEV might have given a reasonable answer. One was simply incorrect in how it was formulated. There was good code already to recursively evaluate the constant offsets in GEPs, look through pointer casts, etc. I gathered this into a form code like the SLP code can use in a previous commit, which allows all of this code to become quite simple. There is some performance (compile time) concern here at first glance as we're directly attempting to walk both pointers constant GEP chains. However, a couple of thoughts: 1) The very common cases where there is a dynamic pointer, and a second pointer at a constant offset (usually a stride) from it, this code will actually not do any unnecessary work. 2) InstCombine and other passes work very hard to collapse constant GEPs, so it will be rare that we iterate here for a long time. That said, if there remain performance problems here, there are some obvious things that can improve the situation immensely. Doing a vectorizer-pass-wide memoizer for each individual layer of pointer values, their base values, and the constant offset is likely to be able to completely remove redundant work and strictly limit the scaling of the work to scrape these GEPs. Since this optimization was not done on the prior version (which would still benefit from it), I've not done it here. But if folks have benchmarks that slow down it should be straight forward for them to add. I've added a test case, but I'm not really confident of the amount of testing done for different access patterns, strides, and pointer manipulation. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@189007 91177308-0d34-0410-b5e6-96231b3b80d8
* Use attribute helper functionMatt Arsenault2013-08-211-2/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188916 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix typoMatt Arsenault2013-08-211-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188915 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix invalid iterator errorsArnold Schwaighofer2013-08-201-13/+51
| | | | | | | | | | | Update iterator when the SLP vectorizer changes the instructions in the basic block by restarting the traversal of the basic block. Patch by Yi Jiang! Fixes PR 16899. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188832 91177308-0d34-0410-b5e6-96231b3b80d8
* PR 16899: Do not modify the basic block using the iterator, but keep theJoerg Sonnenberger2013-08-171-4/+6
| | | | | | | | | next value. This avoids crashes due to invalidation. Patch by Joey Gouly. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188605 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix spellingMatt Arsenault2013-08-151-2/+2
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188506 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix PR16797 - Support PHINodes with multiple inputs from the same basic block.Nadav Rotem2013-08-121-0/+11
| | | | | | | | | Do not generate new vector values for the same entries because we know that the incoming values from the same block must be identical. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@188185 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Fix PR16777. PHInodes may use multiple extracted values that ↵Nadav Rotem2013-08-021-10/+13
| | | | | | | | | | come from different blocks. Thanks Alexey Samsonov. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187663 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorier: update the debug location for the new instructions.Nadav Rotem2013-07-291-2/+22
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187363 91177308-0d34-0410-b5e6-96231b3b80d8
* Don't vectorize when the attribute NoImplicitFloat is used.Nadav Rotem2013-07-291-0/+5
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187340 91177308-0d34-0410-b5e6-96231b3b80d8
* Update the commentNadav Rotem2013-07-271-0/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187316 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP Vectorier: Don't vectorize really short chains because they are already ↵Nadav Rotem2013-07-261-2/+6
| | | | | | handled by the SelectionDAG store-vectorizer, which does a better job in deciding when to vectorize. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187267 91177308-0d34-0410-b5e6-96231b3b80d8
* SLP Vectorizer: Disable the vectorization of non power of two chains, such ↵Nadav Rotem2013-07-261-16/+0
| | | | | | as <3 x float>, because we dont have a good cost model for these types. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@187265 91177308-0d34-0410-b5e6-96231b3b80d8
* When we vectorize across multiple basic blocks we may vectorize PHINodes ↵Nadav Rotem2013-07-221-3/+31
| | | | | | that create a cycle. We already break the cycle on phi-nodes, but arithmetic operations are still uplicated. This patch adds code that checks if the operation that we are vectorizing was vectorized during the visit of the operands and uses this value if it can. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186883 91177308-0d34-0410-b5e6-96231b3b80d8
* Delete unused helper functions.Nadav Rotem2013-07-221-32/+0
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186808 91177308-0d34-0410-b5e6-96231b3b80d8
* Revert a part of r186420. Don't forbid multiple store chains that merge.Nadav Rotem2013-07-211-3/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186786 91177308-0d34-0410-b5e6-96231b3b80d8
* fix an 80-col line.Nadav Rotem2013-07-191-2/+2
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186733 91177308-0d34-0410-b5e6-96231b3b80d8
* Use LLVMs ADTs that improve the compile time of this pass.Nadav Rotem2013-07-191-2/+2
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186732 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Improve the compile time of isConsecutive by reordering the ↵Nadav Rotem2013-07-191-22/+19
| | | | | | conditions that check GEPs and eliminate two of the calls to accumulateConstantOffset. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186731 91177308-0d34-0410-b5e6-96231b3b80d8
* Handle constants without going through SCEV.Nadav Rotem2013-07-181-0/+6
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186593 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Speedup isConsecutive by manually checking GEPs with multiple ↵Nadav Rotem2013-07-181-4/+12
| | | | | | | | | | indices. This brings the compile time of the SLP-Vectorizer to about 2.5% of OPT for my testcase. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186592 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Speedup isConsecutive (that checks if two addresses are ↵Nadav Rotem2013-07-181-12/+31
| | | | | | consecutive in memory) by checking for additional patterns that don't need to go through SCEV. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186563 91177308-0d34-0410-b5e6-96231b3b80d8
* Fix a comment.Nadav Rotem2013-07-171-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186541 91177308-0d34-0410-b5e6-96231b3b80d8
* Add a micro optimization to catch cases where the PtrA equals PtrB.Nadav Rotem2013-07-171-1/+1
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186531 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Accelerate the isConsecutive check by replacing the ↵Nadav Rotem2013-07-171-10/+5
| | | | | | subtraction of the two values with a simple SCEV expression that adds the offset to one of the pointers that we compare. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186479 91177308-0d34-0410-b5e6-96231b3b80d8
* flip the scev minus direction to simplify the code.Nadav Rotem2013-07-161-3/+3
| | | | git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186466 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Improve the compile time of isConsecutive by adding a simple ↵Nadav Rotem2013-07-161-0/+18
| | | | | | | | | | constant-gep check before using SCEV. This check does not always work because not all of the GEPs use a constant offset, but it happens often enough to reduce the number of times we use SCEV. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186465 91177308-0d34-0410-b5e6-96231b3b80d8
* SLPVectorizer: Reduce the compile time of the consecutive store lookup.Nadav Rotem2013-07-161-5/+13
| | | | | | | | Process groups of stores in chunks of 16. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186420 91177308-0d34-0410-b5e6-96231b3b80d8
* PR16628: Fix a bug in the code that merges compares.Nadav Rotem2013-07-151-1/+3
| | | | | | | | Compares return i1 but they compare different types. git-svn-id: https://llvm.org/svn/llvm-project/llvm/trunk@186359 91177308-0d34-0410-b5e6-96231b3b80d8
generated by cgit v1.1 at 2025-11-11 09:57:21 +0000