### A Pluto.jl notebook ###
# v0.16.0

using Markdown
using InteractiveUtils

# ╔═╡ 10349518-03ca-11eb-09b2-69c80c4662ac
begin
	using Pkg
	cd(joinpath(dirname(@__FILE__),".."))
    Pkg.activate(pwd())
	using JuMP, Tulip, GLPK, LinearAlgebra
	using Distributions, Plots, StatsPlots, LaTeXStrings, Measures
end

# ╔═╡ f7f3a256-03c6-11eb-2c1e-83cc62bf55e6
md"""
# Linear programming
We will be using [JuMP](https://jump.dev/JuMP.jl/stable/) as a general framework, combined with [Tulip](https://github.com/ds4dm/Tulip.jl) or [GLPK](https://github.com/jump-dev/GLPK.jl) as a solver.
"""

# ╔═╡ 7910d53e-03ca-11eb-1536-7fc4c236e10a
md"""
## Application - Employee planning
We manage a crew of call center employees and want to optimise our shifts in order to reduce the total payroll cost. Employees have to work for five consecutive days and are then given two days off. The current policy is simple: each day gets the same amount of employees (currently we have 5 persons per shift, which leads to 25 persons on any given day).

We have some historical data that gives us the minimum amount of calls we can expect: Mon: 22, Tue: 17, Wed:13, Thu:14, Fri: 15, Sat: 18, Sun: 24

Employees are payed € 96 per day of work. This lead to the current payroll cost of 25x7x96 = € 16.800. You need to optimize employee planning to reduce the payroll cost.

| Schedule | Days worked | Attibuted Pers | Mon | Tue | Wed | Thu | Fri | Sat | Sun |
|----------|-------------|----------------|-----|-----|-----|-----|-----|-----|-----|
| A | Mon-Fri | 5 | W | W | W | W | W | O | O |
| B | Tue-Sat | 5 | O | W | W | W | W | W | O |
| C | Wed-Sun | 5 | O | O | W | W | W | W | W |
| D | Thu-Mon | 5 | W | O | O | W | W | W | W |
| E | Fri-Tue | 5 | W | W | O | O | W | W | W |
| F | Sat-Wed | 5 | W | W | W | O | O | W | W |
| G | Sun-Thu | 5 | W | W | W | W | O | O | W |
| Totals: | - | 35 | 5 | 5 | 5 | 5 | 5 | 5 | 5 |
| Required: | - | - | 22 | 17 | 13 | 14 | 15 | 18 | 24 |

### Mathematical formulation
We need do formaly define our decision variables, constraints and objective function.
* decision variables: the amount of persons attributed to each schedule ( ``Y = [y_1,y_2,\dots,y_7]^{\intercal} ``)
* objective function: the payroll cost

  Suppose the matrix ``A`` is the matrix indicating the workload for each schedule (in practice ``W=1`` and ``O=0``):
```math
A = \begin{bmatrix}
W & W & W & W & W & O & O \\
O & W & W & W & W & W & O \\
O & O & W & W & W & W & W \\
W & O & O & W & W & W & W \\
W & W & O & O & W & W & W \\
W & W & W & O & O & W & W \\
W & W & W & W & O & O & W 	\\
\end{bmatrix}
```

  Now $$Y*A'$$ gives us a vector indicating the amount of employees working on a given day. Suppose we also use the vector $$c$$ to indicate the salary for a given day (in this case $$c = [96,96,96,\dots,96]$$).

We are now able to write our objective function:
```math
\min Z = c^\intercal A^\intercal Y
```

* constraints (1): each day we need at least enough employees to cover all incoming calls. Suppose we use the vector $$b$$ to indicate the amount of incoming calls for a given day. We are able to write the constraints in a compact way:

```math
\text{subject to } A^\intercal Y  \ge b
```

* constraints (2): we also want to avoid a negative amount of attributed employees on any given day, since this would lead to a negative payroll cost:
```math
\text{and }Y \ge 0
```

* $\forall Y : Y \in \mathbb{N}$
### Implementation
"""

# ╔═╡ 572d14c4-03cb-11eb-2b68-234b3d7e9e8e
begin
	# basic data
	A = ones(Bool,7,7) - diagm(-1=>ones(Bool,6), -2=> ones(Bool,5), 5=>ones(Bool,2), 6=>ones(Bool,1))
	Y = [5,5,5,5,5,5,5]
	B = [22,17,13,14,15,18,24]
	C = [96,96,96,96,96,96,96];
	A
end

# ╔═╡ 4ba4687e-087f-11eb-1cb0-2581818cbd93
# A' * Y .> B
C' * A' * Y

# ╔═╡ 79351ee4-087f-11eb-147d-d389de200857
let
	model = Model(GLPK.Optimizer)
	@variable(model, Y[1:7] >= 0, Int)
	@constraint(model, A' * Y .>= B)
	@objective(model, Min, C' * A' * Y)
	optimize!(model)
	termination_status(model)
	objective_value(model)
	#A' * value.(Y) .> B
	sum(value.(Y))
end

# ╔═╡ 3bafd152-6724-4847-a79d-f2340e2ab5e4

# ╔═╡ 4bc4c2aa-04b4-11eb-2b6e-452d4ecc258a
md"""
### Adding uncertainty
Up to now, we have had constant numbers the minimum number of employees needed per day. In reality these quantities are uncertain. The actual number of calls will fluctuate each day. For simplicity's sake will we use a [lognormal distribution](https://en.wikipedia.org/wiki/Log-normal_distribution#Occurrence_and_applications) for the amount of calls (using their initial value as mean and a standard deviation of two). Working this way, we avoid having negative calls.
"""

# ╔═╡ 6e48e306-04b4-11eb-2561-0151a5e0a908
B

# ╔═╡ 7a54f9b4-04b4-11eb-3a7c-8d90eb026392
begin
	# generating the distributions
	B_u = Distributions.LogNormal.(B,2) # array with distributions

	# quick sample to illustrate amount of calls being randomized
	log.(rand.(B_u))
end

# ╔═╡ 9d3ceafe-0880-11eb-36f0-0f8530d6b285
begin
	cost = Float64[]
	for _ in 1:10000
		let cout=cost
			model = Model(GLPK.Optimizer)
			@variable(model, Y[1:7] >= 0, Int)
			@constraint(model, A' * Y .>= log.(rand.(B_u)))
			@objective(model, Min, C' * A' * Y)
			optimize!(model)
			push!(cout, objective_value(model))
		end
	end
	cost
end

# ╔═╡ 14a32a20-0881-11eb-1a20-715b99417266
StatsPlots.histogram(cost, yscale=:log10)

# ╔═╡ 259bbbc6-04b5-11eb-1ad4-c567e45ba4b6
md"""
### Small variant: adding a commission
Suppose each worker receives extra pay for the amount of calls that have been treated. We can easily include this in our model
"""

# ╔═╡ 83130656-0881-11eb-2c87-a3ae5f81f04b
begin
	cost_c = Float64[]
	commission = 20
	for _ in 1:1000
		let cout=cost_c
			model = Model(GLPK.Optimizer)
			appels = log.(rand.(B_u))
			@variable(model, Y[1:7] >= 0, Int)
			@constraint(model, A' * Y .>= appels)
			@objective(model, Min, C' * A' * Y + sum(appels) * commission)
			optimize!(model)
			push!(cout, objective_value(model))
		end
	end
	StatsPlots.histogram(cost_c)
end

# ╔═╡ e1e41ea6-04b5-11eb-174e-1d43f601a07c
md"""
#### Playing it safe
The above has given us some information on what the distributions of the payroll cost may be, however in reality, you would want to make sure that the clients calling to center are taken care off. To realise this, one might say that for any given day, you want to make sure that 90% of all calls can be treated by the specific capacity.
"""

# ╔═╡ 62b516be-0882-11eb-094b-d183f8d00ab8
log.(quantile.(B_u, 0.90))

# ╔═╡ 6274be66-0882-11eb-11be-05163cda6633
 let
	model = Model(GLPK.Optimizer)
	@variable(model, Y[1:7] >= 0, Int)
	@constraint(model, A' * Y .>= log.(quantile.(B_u, 0.99)))
	@objective(model, Min, C' * A' * Y)
	optimize!(model)
	termination_status(model)
	objective_value(model)
end

# ╔═╡ 1f177098-04b6-11eb-2508-6bd8d7e1e996
md"""
### Additional questions
* The example we have treated so far has very traditional working patterns for the employees. How woud you deal with modern working patterns (e.g. 4/5 or parttime working)?
* We took a look at the stochastic nature of the amount of calls, however, the personnel might not show up for various reasons. How would you describe the possible influence? Hint: make a discrete event model of this setting, using the optimal design and controlling for employees showing up or not.
"""

# ╔═╡ Cell order:
# ╟─f7f3a256-03c6-11eb-2c1e-83cc62bf55e6
# ╠═10349518-03ca-11eb-09b2-69c80c4662ac
# ╟─7910d53e-03ca-11eb-1536-7fc4c236e10a
# ╠═572d14c4-03cb-11eb-2b68-234b3d7e9e8e
# ╠═4ba4687e-087f-11eb-1cb0-2581818cbd93
# ╠═79351ee4-087f-11eb-147d-d389de200857
# ╠═3bafd152-6724-4847-a79d-f2340e2ab5e4
# ╟─4bc4c2aa-04b4-11eb-2b6e-452d4ecc258a
# ╠═6e48e306-04b4-11eb-2561-0151a5e0a908
# ╠═7a54f9b4-04b4-11eb-3a7c-8d90eb026392
# ╠═9d3ceafe-0880-11eb-36f0-0f8530d6b285
# ╠═14a32a20-0881-11eb-1a20-715b99417266
# ╟─259bbbc6-04b5-11eb-1ad4-c567e45ba4b6
# ╠═83130656-0881-11eb-2c87-a3ae5f81f04b
# ╟─e1e41ea6-04b5-11eb-174e-1d43f601a07c
# ╠═62b516be-0882-11eb-094b-d183f8d00ab8
# ╠═6274be66-0882-11eb-11be-05163cda6633
# ╟─1f177098-04b6-11eb-2508-6bd8d7e1e996

Exercises/PS05 - Linear programming I.jl


module Bisect

export idx_of_closest
#export bin_search, idx_of_closest

function midpoint(a, b)
    if a > b
        error("... b must be greater or equal than a; a == $a, b == $b")
    elseif a == b
        a
    else
        div(a + b , 2)
    end
end

#................ 1) Search bin
function search_bin(edges::Vector{Float64}, val::Float64, imin::Int, imax::Int)
    # find the BIN - characterized by the pair of edges
    # If
    imid = midpoint(imin, imax)
    if imin == imax
        if val >= edges[imid]
            imid + 1
        else
            imid
        end
    else
        if val >= edges[imid]
            search_bin(edges, val, imid + 1, imax)
        else
            search_bin(edges, val, imin, max(imid - 1, imin))
        end
    end
end

function search_bin(edges::Vector{Float64}, val::Float64)
    search_bin(edges, val, 1, length(edges))
end

function test_search_bin()
    println( search_bin([0.1, 0.2, 0.3], 0.05)) # 1
    println( search_bin([0.1, 0.2, 0.3], 0.10)) # 2
    println( search_bin([0.1, 0.2, 0.3], 0.15)) # 2
    println( search_bin([0.1, 0.2, 0.3], 0.19)) # 2
    println( search_bin([0.1, 0.2, 0.3], 0.29)) # 3
    println( search_bin([0.1, 0.2, 0.3], 0.31)) # 4
end
# test_search_bin()
#................ End 1) Search bin
#............... 2) Index of the closest element
function idx_of_closest(edges::Vector{Float64}, val::Float64, imin::Int, imax::Int)
    # Find index of the closest edge to the value
    bin  = search_bin(edges, val)
    n = length(edges)
    if bin == 1
        edges[bin]
    elseif bin == n + 1
        edges[n]
    else
        left_dist  = abs(val - edges[bin-1])
        right_dist = abs(val - edges[bin]  )
        left_dist >= right_dist ? bin : bin - 1
    end
end

function idx_of_closest(edges::Vector{Float64}, val::Float64)
    idx_of_closest(edges, val, 1, length(edges))
end

function test_bin_search_closest_idx()
    println( idx_of_closest([0.1, 0.2, 0.5], 0.15))
    println( idx_of_closest([0.1, 0.2, 0.5], 0.16))
end
#test_bin_search_closest_idx()
#............... End 2) Index of the closest element
end


"""
hatfun(x,t,k)

Evaluate a piecewise linear "hat" function at `x`, where `t` is a
vector of n+1 interpolation nodes and `k` is an integer in 0:n
giving the index of the node where the hat function equals one.
"""

function hatfun(x,t,k)
    n = length(t)-1
    # Return correct node given mathematical index k, including
    # fictitious choices.
    function node(k)
        if k < 0
            2t[1]-t[2]
        elseif k > n
            2t[n+1]-t[n]
        else
            t[k+1]
        end
    end

    H1 = (x-node(k-1))/(node(k)-node(k-1))   # upward slope
    H2 = (node(k+1)-x)/(node(k+1)-node(k))   # downward slope

    H = min(H1,H2)
    return max(0,H)
end

"""
plinterp(t,y)

Create a piecewise linear interpolating function for data values in
`y` given at nodes in `t`.
"""
function plinterp(t,y)
n = length(t)-1
return x -> sum( y[k+1]*hatfun(x,t,k) for k in 0:n )
end

"""
spinterp(t,y)

Create a cubic not-a-knot spline interpolating function for data
values in `y` given at nodes in `t`.
"""
function spinterp(t,y)

    n = length(t)-1
    h = diff(t)         # differences of all adjacent pairs

    # Preliminary definitions.
    Z = zeros(n,n);
    In = I(n);  E = In[1:n-1,:];
    J = diagm(0=>ones(n),1=>-ones(n-1))
    H = diagm(0=>h)

    # Left endpoint interpolation:
    AL = [ In Z Z Z ]
    vL = y[1:n]

    # Right endpoint interpolation:
    AR = [ In H H^2 H^3 ];
    vR = y[2:n+1]

    # Continuity of first derivative:
    A1 = E*[ Z J 2*H 3*H^2 ]
    v1 = zeros(n-1)

    # Continuity of second derivative:
    A2 = E*[ Z Z J 3*H ]
    v2 = zeros(n-1)

    # Not-a-knot conditions:
    nakL = [ zeros(1,3*n) [1 -1 zeros(1,n-2)] ]
    nakR = [ zeros(1,3*n) [zeros(1,n-2) 1 -1] ]

    # Assemble and solve the full system.
    A = [ AL; AR; A1; A2; nakL; nakR ]
    v = [ vL; vR; v1; v2; 0; 0 ]
    z = A\v

    # Break the coefficients into separate vectors.
    rows = 1:n
    a = z[rows]
    b = z[n.+rows];  c = z[2*n.+rows];  d = z[3*n.+rows]
    S = [ Polynomial([a[k],b[k],c[k],d[k]]) for k = 1:n ]
    # This function evaluates the spline when called with a value
    # for x.
    function evaluate(x)
        k = findfirst(@. x<t)   # one greater than interval x belongs to
        k==1 && return NaN
        if isnothing(k)
            return x==t[end] ? y[end] : NaN
        end
        return S[k-1](x-t[k-1])
    end
    return evaluate
end

"""
fdweights(t,m)

Return weights for the `m`th derivative of a function at zero using
values at the nodes in vector `t`.
"""
function fdweights(t,m)
    # This is a compact implementation, not an efficient one.

    function weight(t,m,r,k)
        # Recursion for one weight. Input: t   nodes (vector) m
        # order of derivative sought r   number of nodes to use from
        # t (<= length(t)) k   index of node whose weight is found

        if (m<0) || (m>r)        # undefined coeffs must be zero
            c = 0
        elseif (m==0) && (r==0)  # base case of one-point interpolation
            c = 1
        else                     # generic recursion
            if k<r
                c = (t[r+1]*weight(t,m,r-1,k) -
                    m*weight(t,m-1,r-1,k))/(t[r+1]-t[k+1])
            else
                numer = r > 1 ? prod(t[r]-x for x in t[1:r-1]) : 1.0
                denom = r > 0 ? prod(t[r+1]-x for x in t[1:r]) : 1.0
                β = numer/denom
                c = β*(m*weight(t,m-1,r-1,r-1) - t[r]*weight(t,m,r-1,r-1))
            end
        end
        return c
    end

    r = length(t)-1
    w = zeros(size(t))
    return [ weight(t,m,r,k) for k=0:r ]
end

"""
trapezoid(f,a,b,n)

Apply the trapezoid integration formula for integrand `f` over
interval [`a`,`b`], broken up into `n` equal pieces. Returns
estimate, vector of nodes, and vector of integrand values at the
nodes.
"""
function trapezoid(f,a,b,n)
    h = (b-a)/n
    t = LinRange(a,b,n+1)
    y = f.(t)
    T = h * ( sum(y[2:n]) + 0.5*(y[1] + y[n+1]) )

    return T,t,y
end

"""
intadapt(f,a,b,tol)

Do adaptive integration to estimate the integral of `f` over
[`a`,`b`] to desired error tolerance `tol`. Returns estimate and a
vector of evaluation nodes used.
"""
function intadapt(f,a,b,tol)
    # Use error estimation and recursive bisection.
    function do_integral(a,fa,b,fb,m,fm,tol)
        # These are the two new nodes and their f-values.
        xl = (a+m)/2;  fl = f(xl);
        xr = (m+b)/2;  fr = f(xr);
        t = [a,xl,m,xr,b]          # all 5 nodes at this level

        # Compute the trapezoid values iteratively.
        h = (b-a)
        T = [0.,0.,0.]
        T[1] = h*(fa+fb)/2
        T[2] = T[1]/2 + (h/2)*fm
        T[3] = T[2]/2 + (h/4)*(fl+fr)

        S = (4*T[2:3]-T[1:2]) / 3      # Simpson values
        E = (S[2]-S[1]) / 15           # error estimate

        if abs(E) < tol*(1+abs(S[2]))  # acceptable error?
            Q = S[2]                   # yes--done
        else
            # Error is too large--bisect and recurse.
            QL,tL = do_integral(a,fa,m,fm,xl,fl,tol)
            QR,tR = do_integral(m,fm,b,fb,xr,fr,tol)
            Q = QL + QR
            t = [tL;tR[2:end]]   # merge the nodes w/o duplicate
        end
        return Q,t
    end

    m = (b+a)/2
    Q,t = do_integral(a,f(a),b,f(b),m,f(m),tol)
    return Q,t
end


"""
    forwardsub(L,b)

Solve the lower-triangular linear system with matrix `L` and right-hand side
vector `b`.
"""
function forwardsub(L,b)

    n = size(L,1)
    x = zeros(n)
    for i = 1:n
        s = i > 1 ? sum( L[i,j]*x[j] for j=1:i-1 ) : 0
        x[i] = ( b[i] - s ) / L[i,i]
    end

    return x
end

"""
    backsub(U,b)

Solve the upper-triangular linear system with matrix `U` and right-hand side
vector `b`.
"""
function backsub(U,b)

    n = size(U,1)
    x = zeros(n)
    for i = n:-1:1
        s = i < n ? sum( U[i,j]*x[j] for j=i+1:n ) : 0
        x[i] = ( b[i] - s ) / U[i,i]
    end

    return x
end

"""
    lufact(A)

Compute the LU factorization of square matrix `A`, returning the factors.
"""
function lufact(A)

    n = size(A,1)
    L = Matrix(Diagonal(ones(n)))    # puts ones on diagonal
    U = copy(A)

    # Gaussian elimination
    for j = 1:n-1
        for i = j+1:n
            L[i,j] = U[i,j] / U[j,j]   # row multiplier
            U[i,j:n] = U[i,j:n] - L[i,j]*U[j,j:n]
        end
    end

    return L,triu(U)
end


using LinearAlgebra

function classical_gram_schmidt(A)
    m, n = size(A)
    R = zeros(n, n)
    Q = zeros(m, n)
    for j in 1:n
        v_j = A[:, j]
        for i in 1:(j-1)
            R[i, j] = dot(Q[:, i], A[:, j])
            v_j = v_j - R[i, j] * Q[:, i]
        end
        R[j, j] = norm(v_j)
        Q[:, j] = v_j / R[j, j]
    end
    return Q, R
end

function modified_gram_schmidt(A)
    m, n = size(A)
    R = zeros(Float64, n, n)
    Q = zeros(Float64, m, n)

    V = zeros(Float64, n, n)
    for i in 1:n
        V[:, i] = A[:, i]
    end

    for i in 1:n
        R[i, i] = norm(V[:, i])
        Q[:, i] = V[:, i] / R[i, i]
        for j in (i+1):n
            R[i, j] = dot(Q[:, i], V[:, j])
            V[:, j] = V[:, j] - R[i, j] * Q[:, i]
        end
    end
    return Q, R
end

function check_stability(Q)
    m, n = size(Q)
    total_dot_error = 0
    for i in 1:m
        for j in 1:n
            if i != j
                total_dot_error += abs(dot(Q[:, i], Q[:, j]))
            end
        end
    end
    return total_dot_error
end


"""
Estimate if the number is prime
"""
function check_composite(a::BigInt, d::BigInt, n::BigInt, s::BigInt)
    x = exp!(a, d, n)

    if (x == 1 || x == (n - 1))
        return (false)
    end

    for _ in 1:s
        x = (x * x) % n
        if (x == (n - 1))
            return (false)
        end
    end
    true
end

check_composite(a::T, d::T, n::T, s::T) where {T <: Integer} =
check_composite(big(a), big(d), big(n), big(s))

"""Miller Rabin primality test
Return False if n is composite, True(probably prime) otherwise.

n: Number to be tested for primality
k: number of iterations to run the test
"""
function miller_rabin(n::BigInt, k::BigInt = big(10))
    if (n == 2)
        return (true)
    end
    if (n == 1 || n % 2 == 0)
        return (false)
    end

    s::BigInt = 0
    d = n - 1

    while d % 2 == 0
        s += 1
        d /= 2
    end

    @assert floor(BigInt, 2 ^ s * d) == n - 1

    for _ in 1:k
        a = rand(2:n - 1)
        d = floor(BigInt, d)
        if (check_composite(a, d, n, s))
            return (false)
        end
    end
    true
end

miller_rabin(n::T, k::T = 10) where {T <: Integer} =
miller_rabin(big(n), big(k))

"""
Solovay Strassen Primality Test
Returns False is n is composite, True(probably prime) otherwise

n: Number to be tested for primality
k: number of iterations to run the test
"""
function solovay_strassen(n::BigInt, k::BigInt = big(10))
    if (n == 2)
        return (true)
    end
    if (n == 1 || n & 1 == 0)
        return (false)
    end

    for _ in 1:k
        a = rand(2:n-1)
        x = (jacobi!(a, n) + n) % n
        if (x == 0 || exp!(a, floor(BigInt, (n - 1) / 2), n) != x)
            return (false)
        end
    end
    true
end

solovay_strassen(n::T, k::T = 10) where {T <: Integer} =
solovay_strassen(big(n), big(k))

"""
Fermat's Primality test
Returns True(probably prime) if n is prime, Flase if n is composite

n: Number to be tested for primality
k: number of iterations of the Fermat's Primality Test
"""
function fermat_test(n::BigInt, k::BigInt = big(10))
    if (n == 2)
        return (true)
    end
    if (n & 1 == 0)
        return (false)
    end

    for _ in 1:k
        a = rand(2:n-1)
        if (exp!(a, n - 1, n) != 1)
            return (false)
        end
    end
    true
end

fermat_test(n::T, k::T = 10) where {T <: Integer} =
fermat_test(big(n), big(k))


export SoftMax

"""
`SoftMax([domainType=Float64::Type,] dim_in::Tuple)`

Creates the softmax non-linear operator with input dimensions `dim_in`.
```math
\\sigma(\\mathbf{x}) = \\frac{e^{\\mathbf{x} }}{ \\sum e^{\\mathbf{x} } }
```

"""
struct SoftMax{T,N} <: NonLinearOperator
	dim::NTuple{N,Int}
	buf::Array{T,N}
end

function SoftMax(DomainType::Type, DomainDim::NTuple{N,Int}) where {N}
	SoftMax{DomainType,N}(DomainDim,zeros(DomainType,DomainDim))
end

SoftMax(DomainDim::NTuple{N,Int}) where {N} = SoftMax(Float64,DomainDim)

function mul!(y::AbstractArray{T,N}, L::SoftMax{T,N}, x::AbstractArray{T,N}) where {T,N}
	y .= exp.(x.-maximum(x))
	y ./= sum(y)
end

function mul!(y::AbstractArray,
              J::AdjointOperator{Jacobian{A,TT}},
              b::AbstractArray) where {T, N, A<: SoftMax{T,N}, TT <: AbstractArray{T,N} }
    L = J.A
	fill!(y,zero(T))
	L.A.buf .= exp.(L.x.-maximum(L.x))
	L.A.buf ./= sum(L.A.buf)
	for i in eachindex(y)
		y[i] = -L.A.buf[i]*dot(L.A.buf,b)
		y[i] += L.A.buf[i]*b[i]
	end
	return y
end

fun_name(L::SoftMax) = "σ"

size(L::SoftMax) = (L.dim, L.dim)

domainType(L::SoftMax{T,N}) where {T,N} = T
codomainType(L::SoftMax{T,N}) where {T,N} = T


@doc doc"""
Approximation de la solution du problème ``\min_{x \in \mathbb{R}^{n}} f(x)`` en utilisant l'algorithme de Newton

# Syntaxe
```julia
xk,f_min,flag,nb_iters = Algorithme_de_Newton(f,gradf,hessf,x0,option)
```

# Entrées :
   * **f**       : (Function) la fonction à minimiser
   * **gradf**   : (Function) le gradient de la fonction f
   * **hessf**   : (Function) la Hessienne de la fonction f
   * **x0**      : (Array{Float,1}) première approximation de la solution cherchée
   * **options** : (Array{Float,1})
       * **max_iter**      : le nombre maximal d'iterations
       * **Tol_abs**       : la tolérence absolue
       * **Tol_rel**       : la tolérence relative

# Sorties:
   * **xmin**    : (Array{Float,1}) une approximation de la solution du problème  : ``\min_{x \in \mathbb{R}^{n}} f(x)``
   * **f_min**   : (Float) ``f(x_{min})``
   * **flag**     : (Integer) indique le critère sur lequel le programme à arrêter
      * **0**    : Convergence
      * **1**    : stagnation du xk
      * **2**    : stagnation du f
      * **3**    : nombre maximal d'itération dépassé
   * **nb_iters** : (Integer) le nombre d'itérations faites par le programme

# Exemple d'appel
```@example
using Optinum
f(x)=100*(x[2]-x[1]^2)^2+(1-x[1])^2
gradf(x)=[-400*x[1]*(x[2]-x[1]^2)-2*(1-x[1]) ; 200*(x[2]-x[1]^2)]
hessf(x)=[-400*(x[2]-3*x[1]^2)+2  -400*x[1];-400*x[1]  200]
x0 = [1; 0]
options = []
xmin,f_min,flag,nb_iters = Algorithme_De_Newton(f,gradf,hessf,x0,options)
```
"""
function Algorithme_De_Newton(f::Function,gradf::Function,hessf::Function,x0,options)

    "# Si options == [] on prends les paramètres par défaut"
    if options == []
        max_iter = 100
        Tol_abs = sqrt(eps())
        Tol_rel = 1e-15
    else
        max_iter = options[1]
        Tol_abs = options[2]
        Tol_rel = options[3]
    end

    n = length(x0)
    xmin = x0
    f_min = 0
    flag = 0
    nb_iters = 0
    xk = x0
    xk_1 = x0

    while (norm(gradf(xk)) > max(Tol_rel * norm(gradf(x0)), Tol_abs))
        nb_iters += 1
        xk_1 = xk
        dk = -hessf(xk) \ gradf(xk)
        xk = xk + dk

        if norm(gradf(xk_1)) <= max(Tol_abs, Tol_rel * norm(gradf(x0)))
            flag = 0
            break

        elseif norm(dk) <= max(Tol_abs, Tol_rel * norm(xk))
            flag = 1
            break

        elseif abs(f(xk_1) - f(xk)) <= max(Tol_abs, Tol_rel * abs(f(xk)))
            flag = 2
            break
        elseif nb_iters == max_iter
            flag = 3
            break
        end
    end

    xmin = xk
    f_min = f(xmin)

    return xmin,f_min,flag,nb_iters
end


function Example_4_3(; T = 200, seed = 1)

    # Set random number generator

    rg = MersenneTwister(seed)

    # state x is four dimensional, comprises car position and velocities

    Δt  = 1 / 10
    σ₁  = σ₂  = 1 / 2
    qᶜ₁ = qᶜ₂ = 1

    # Matrices in the dynamic model are:

    A = [1  0  Δt  0;
         0  1   0  Δt;
         0  0   1  0;
         0  0   0  1]

    Q = [(qᶜ₁*Δt^3)/3         0        (qᶜ₁*Δt^2)/2       0        ;
              0         (qᶜ₂*Δt^3)/3         0        (qᶜ₁*Δt^2)/2 ;
         (qᶜ₁*Δt^2)/2         0            qᶜ₁*Δt         0        ;
              0         (qᶜ₂*Δt^2)/2         0          qᶜ₂*Δt     ]

    # Measurement matrices

    H = [1 0 0 0;
         0 1 0 0]

    R = [σ₁^2  0;
         0    σ₂^2]

    # initialise

    m₀, P₀ = zeros(4), 0.1*Matrix(I, 4, 4)

    x₀ = rand(MvNormal(m₀, P₀))

    # store measurements and states

    yclean = Array{Vector{Float64},1}(undef, T)

    y = Array{Vector{Float64},1}(undef, T)

    x = Array{Vector{Float64},1}(undef, T)

    # first state

    x[1] = A * x₀

    y[1] = rand(rg, MvNormal(H * x[1], R))

    for k in 2:T

        x[k] = rand(rg, MvNormal(A * x[k-1], Q)) # (4.32)

        yclean[k] = H * x[k]

        y[k] = rand(rg, MvNormal(H * x[k], R)) # (4.33)

    end

    return yclean, y, x, A, H, Q, R, m₀, P₀

end

function runExample_4_3(seed = 1)

    # Simulate data

    yclean, y, x, A, H, Q, R, m₀, P₀ = Example_4_3(seed=seed)

    # Inference

    μfilter, Σfilter = filteringrecursion(y; A = A, H = H, Q = Q, R = R, m₀ = m₀, P₀ = P₀)

    μsmooth, Σsmooth = smoothingrecursion(y; A = A, H = H, Q = Q, R = R, m = μfilter, P = Σfilter)

    # collect positions

    truepositions = reduce(hcat, [xₖ[1:2] for xₖ in x])

    filtpositions = reduce(hcat, [xₖ[1:2] for xₖ in μfilter])

    smoopositions = reduce(hcat, [xₖ[1:2] for xₖ in μsmooth])

    # plot positions

    figure(1)

    subplot(211); cla()

    plot(truepositions[1,:], truepositions[2,:], "o", label = "true positions")

    plot(filtpositions[1,:], filtpositions[2,:], ".", label = "filter positions", alpha=0.5)

    legend()

    subplot(212); cla()

    plot(truepositions[1,:], truepositions[2,:], "o", label = "true positions")

    plot(smoopositions[1,:], smoopositions[2,:], ".", label = "smooth positions", alpha=0.5)

    legend()

    nothing

end


function Primes(N::Int64)

	isprime = ones(Bool,N)

	for i = 2:int64(sqrt(N))
		if isprime[i]
			for j in (i*i):i:N
				isprime[j] = false
			end
		end
	end
	return filter(x -> isprime[x], 2:N)
end

function TrialDivision(N::Int64)

	primeFactors = [1]

	testPrimes = Primes(N);
	Nc = N;
	idx = 1;
	while (prod(primeFactors)!=N) && (idx < length(testPrimes))
		if mod(Nc, testPrimes[idx])==0
			println(testPrimes[idx])
			primeFactors=[primeFactors,testPrimes[idx]]
			Nc = Nc/testPrimes[idx]
		end
		idx = idx+1
	end

	return primeFactors[2:end]
end

function LargestPrimeFactor(N::Int64)

	first20 = Primes(1000)
	largest = 1
	for p in first20
		if mod(N, p) == 0
			largest = p
			N = fld(N,p)
		end
	end
	if N == 1
		return largest
	end
	trials = flipud(Primes(N+1))
	for p in trials
		if (mod(N,p) == 0)&&(p > largest)
			println(p)
			largest = p
			break
		end
	end
	if largest == 1
		return N
	else
		return largest
	end
end

function Euler3(N::Int64)

	big = LargestPrimeFactor(N)
	println("The largest prime factor of ",N," is ",big,".")

end

@time Euler3(600851475143)
#Euler3(100)


"""
    bisection(f,a,b)

adalah fungsi yang mencari nilai akar persamaan dari fungsi `f(x)` pada interval
di antara `a` dan `b`.

# Examples
```jldoctest
julia> f(x) = x*sin(x)-1;

julia> c, flag, M = bisection(f,0,2);

julia> c
1.1141571998596191

julia> flag
0

julia> M
22×5 Array{Float64,2}:
  0.0  0.0      1.0      2.0      -0.158529
  1.0  1.0      1.5      2.0       0.496242
  2.0  1.0      1.25     1.5       0.186231
  3.0  1.0      1.125    1.25      0.015051
  ⋮
 18.0  1.11415  1.11415  1.11416  -3.22926e-6
 19.0  1.11415  1.11416  1.11416  -5.80313e-7
 20.0  1.11416  1.11416  1.11416   7.44159e-7
 21.0  1.11416  1.11416  1.11416   8.19227e-8
```
return solusi `c` dengan `flag` bernilai 0 jika metode bisection berhasil menemukan
solusi dan gagal jika tidak 0. Serta, matriks `M` yang berisi catatan proses tiap
iterasi `[k, a, c, b, f(c)]`

"""
function bisection(f,a,b)
    delta = 10^-7; maxi = 100; flag = 1;
    M = Array{Float64}(undef, 0, 5);
    fa = f(a); fb = f(b);
    if fa*fb>0
        c = "error: f(a) dan f(b) harus berbeda tanda";
        flag = 2;
        return;
    end
    k = 1
    while k<=maxi
        c  = (b+a)/2;
        fc = f(c);
        M = [M; [k-1 a c b fc] ];
        if fc == 0
            a = c;
            b = c;
        elseif fa*fc>0
            a = c;
            fa = fc;
        else
            b = c;
            fb = fc;
        end
        if b-a < delta || abs(fc) < delta
            flag = 0;
            break;
        end
        k+=1
    end
    return c,flag, M
end


using LinearAlgebra
using SparseArrays
using Plots
using ForwardDiff

"This code applies the method of manufactured solutions to the 2D Laplace's equation
with nonzero Dirichlet BCs using a finite difference method."

m = 100 # number of points

uexact(x,y) = log(2+sin(pi*x)*sin(pi*y))
dudx_exact(x,y) = ForwardDiff.derivative(x->uexact(x,y),x)
dudy_exact(x,y) = ForwardDiff.derivative(y->uexact(x,y),y)
du2dx2_exact(x,y) = ForwardDiff.derivative(x->dudx_exact(x,y),x)
du2dy2_exact(x,y) = ForwardDiff.derivative(y->dudy_exact(x,y),y)
f(x,y) = -(du2dx2_exact(x,y) + du2dy2_exact(x,y))

u_left(y)   = uexact(-1,y)
u_right(y)  = uexact(1,y)
u_bottom(x) = uexact(x,-1)
u_top(x)    = uexact(x,1)
uBC = (u_left,u_right,u_bottom,u_top)

function solve(m,f,uBC)
    u_left,u_right,u_bottom,u_top = uBC
    x = LinRange(-1,1,m+2)
    y = LinRange(-1,1,m+2)
    xint = x[2:end-1]
    yint = y[2:end-1]
    h = x[2]-x[1]

    id(i,j,m) = i + (j-1)*m # return global index from i,j

    function FD_matrix_2D(h,m)
        A = spzeros(m*m,m*m)
        for i = 1:m, j = 1:m # loop thru x,y indices
            A[id(i,j,m),id(i,j,m)] = 4/h^2
            if i > 1 # avoids leftmost line of nodes
                A[id(i,j,m),id(i-1,j,m)] = -1/h^2 # x-derivative
            end
            if i < m # avoids rightmost line of nodes
                A[id(i,j,m),id(i+1,j,m)] = -1/h^2
            end
            if j > 1
                A[id(i,j,m),id(i,j-1,m)] = -1/h^2 # y-derivative
            end
            if j < m
                A[id(i,j,m),id(i,j+1,m)] = -1/h^2
            end
        end
        return A
    end

    A = FD_matrix_2D(h,m)

    b = vec(f.(xint,yint'))

    # impose non-zero Dirichlet BCs
    for i = 1:m, j = 1:m
        if i==1
            b[id(i,j,m)] += u_left(yint[j])/h^2
        end
        if i==m
            b[id(i,j,m)] += u_right(yint[j])/h^2
        end
        if j==1
            b[id(i,j,m)] += u_bottom(xint[i])/h^2
        end
        if j==m
            b[id(i,j,m)] += u_top(xint[i])/h^2
        end
    end

    # solve Au = f(xi,yj)
    u = A\b
    return u,h,xint,yint
end

errvec = []
hvec = []
for m in [4 8 16 32 64 128]
    u,h,xint,yint = solve(m,f,uBC)
    err = maximum(abs.(u .- vec(uexact.(xint,yint'))))
    append!(errvec,err)
    append!(hvec,h)
end
plot(hvec,errvec,mark=:dot,xaxis=:log,yaxis=:log)
plot!(hvec,hvec.^2,ls=:dash,xaxis=:log,yaxis=:log)
# contourf(xint,yint,))


#=

The four adjacent digits in the 1000-digit number that have the greatest product are 9 × 9 × 8 × 9 = 5832.

73167176531330624919225119674426574742355349194934
96983520312774506326239578318016984801869478851843
85861560789112949495459501737958331952853208805511
12540698747158523863050715693290963295227443043557
66896648950445244523161731856403098711121722383113
62229893423380308135336276614282806444486645238749
30358907296290491560440772390713810515859307960866
70172427121883998797908792274921901699720888093776
65727333001053367881220235421809751254540594752243
52584907711670556013604839586446706324415722155397
53697817977846174064955149290862569321978468622482
83972241375657056057490261407972968652414535100474
82166370484403199890008895243450658541227588666881
16427171479924442928230863465674813919123162824586
17866458359124566529476545682848912883142607690042
24219022671055626321111109370544217506941658960408
07198403850962455444362981230987879927244284909188
84580156166097919133875499200524063689912560717606
05886116467109405077541002256983155200055935729725
71636269561882670428252483600823257530420752963450

Find the thirteen adjacent digits in the 1000-digit number that have the greatest product. What is the value of this product?

=#

str ="""73167176531330624919225119674426574742355349194934
        96983520312774506326239578318016984801869478851843
        85861560789112949495459501737958331952853208805511
        12540698747158523863050715693290963295227443043557
        66896648950445244523161731856403098711121722383113
        62229893423380308135336276614282806444486645238749
        30358907296290491560440772390713810515859307960866
        70172427121883998797908792274921901699720888093776
        65727333001053367881220235421809751254540594752243
        52584907711670556013604839586446706324415722155397
        53697817977846174064955149290862569321978468622482
        83972241375657056057490261407972968652414535100474
        82166370484403199890008895243450658541227588666881
        16427171479924442928230863465674813919123162824586
        17866458359124566529476545682848912883142607690042
        24219022671055626321111109370544217506941658960408
        07198403850962455444362981230987879927244284909188
        84580156166097919133875499200524063689912560717606
        05886116467109405077541002256983155200055935729725
        71636269561882670428252483600823257530420752963450"""
long_string = replace(str, "\n" => "")
len_adjacent = 13

function largest_adjacent_product(string, len_adjacent)
    num_digits = length(string)
    largest_product = 0
    largest_piece = 0
    for offset in 0:num_digits-len_adjacent
        number_piece = string[1+offset:len_adjacent+offset]
        #println(number_piece)
        product = prod(digits(parse(Int, number_piece)))
        if product > largest_product
            largest_product = product
            largest_piece = number_piece
        end

    end
    return largest_product, largest_piece
end

println()
print(largest_adjacent_product(long_string, len_adjacent))


function sinh(x::DoubleFloat{T}) where {T<:IEEEFloat}
    x < 0 && return -sinh(-x)
    iszero(x) && return zero(x)
    !isfinite(x) && return nan(typeof(x))

    y = exp(x) - exp(-x)
    z = DoubleFloat{T}(y.hi*0.5, y.lo*0.5)
    return z
end

function cosh(x::DoubleFloat{T}) where {T<:IEEEFloat}
    x < 0 && return cosh(-x)
    iszero(x) && return one(x)
    !isfinite(x) && return nan(typeof(x))

    y = exp(x) + exp(-x)
    z = DoubleFloat{T}(y.hi*0.5, y.lo*0.5)
    return z
end

function tanh(x::DoubleFloat{T}) where {T<:IEEEFloat}
    return sinh(x) / cosh(x)
end
#=
function tanh(x::DoubleFloat{T}) where {T<:IEEEFloat}
    x < 0 && return -tanh(-x)
    iszero(x) && return zero(x)
    !isfinite(x) && return nan(typeof(x))
    epos = exp(x)
    eneg = exp(-x)
    z = (epos - eneg) / (epos + eneg)
    return z
end
=#

csch(x::DoubleFloat{T}) where {T<:IEEEFloat} =
    inv(sinh(x))

sech(x::DoubleFloat{T}) where {T<:IEEEFloat} =
    inv(cosh(x))

coth(x::DoubleFloat{T}) where {T<:IEEEFloat} =
    inv(tanh(x))


using Plots

##------------------------------------------------------------------------------

# Ch 7.1 Principles of Regression
# Julia code to fit data points using a straight line
N = 50
x = rand(N)
a = 2.5                         # true parameter
b = 1.3                         # true parameter
y = a*x .+ b + 0.2*rand(N)      # Synthesize training data

X = [x ones(N)]                 # construct the X matrix
θ = X\y                         # solve y = X*θ

t    = range(0,stop=1,length=200)
yhat = θ[1]*t .+ θ[2]                  # fitted values at t

p1 = scatter(x,y,makershape=:circle,label="data",legend=:topleft)
plot!(t,yhat,linecolor=:red,linewidth=4,label="best fit")
display(p1)
##------------------------------------------------------------------------------

# Julia code to fit data using a quadratic equation
N = 50
x = rand(N)
a = -2.5
b = 1.3
c = 1.2
X = x.^[0 1 2]                #same as [ones(N) x x.^2]
y = X*[c,b,a] + rand(N)

θ    = X\y
t    = range(0,stop=1,length=200)
yhat = (t.^[0 1 2])*θ               #same as (t.^collect(0:2)')*θ

p2 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft)
plot!(t,yhat,linecolor=:red,linewidth=4,label="fitted curve")
display(p2)
##------------------------------------------------------------------------------

# Julia code to fit data using Legendre polynomials

using LegendrePolynomials

N = 50
x = rand(N)*2 .- 1
a = [-0.001, 0.01, 0.55, 1.5, 1.2]
X = hcat([Pl.(x,p) for p=0:4]...)   # same as [Pl.(x,0) Pl.(x,1) Pl.(x,2) Pl.(x,3) Pl.(x,4)]
y = X*a + 0.5*randn(N)
θ = X\y

t    = range(-1,stop=1,length=200)
yhat = hcat([Pl.(t,p) for p=0:4]...)*θ

p3 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft)
plot!(t,yhat,linecolor=:red,linewidth=4,label="fitted curve")
display(p3)
##------------------------------------------------------------------------------

# Julia code for auto-regressive model
using ToeplitzMatrices

N = 500
y = cumsum(0.2*randn(N)) + 0.05*randn(N)        # generate data

L = 100                                         # use previous 100 samples
c = [0; y[1:400-1]]
r = zeros(L)
X = Matrix(Toeplitz(c,r))                       # Toeplitz matrix, converted
θ = X\y[1:400]                                  # solve y = X*θ
yhat = X*θ                                      # prediction

p4 = scatter(y[1:400],makershape=:circle,label="data",legend=:bottomleft)
plot!(yhat[1:400],linecolor=:red,linewidth=4,label="fitted curve")
display(p4)
##------------------------------------------------------------------------------

# Julia code to demonstrate robust regression

using LinearAlgebra, LegendrePolynomials, Convex, SCS
import MathOptInterface
const MOI = MathOptInterface

#----------------------------------------------------------
"""
    linprog_Convex(c,A,sense,b)

A wrapper for doing linear programming with Convex.jl/SCS.jl.
It solves, for instance, `c'x` st. `A*x<=b`
"""
function linprog_Convex(c,A,sense,b)
    n  = length(c)
    x  = Variable(n)
    #c1 = sense(A*x,b)             #restriction, for Ax <= b, use sense = (<=)
    c1 = A*x <= b
    prob = minimize(dot(c,x),c1)
    solve!(prob,()->SCS.Optimizer(verbose = false))
    x_i = prob.status == MOI.OPTIMAL ? vec(evaluate(x)) : NaN
    return x_i
end
#----------------------------------------------------------

N = 50
x = range(-1,stop=1,length=N)
a = [-0.001, 0.01, 0.55, 1.5, 1.2]
y = hcat([Pl.(x,p) for p=0:4]...)*a  + 0.05*randn(N)        # generate data
idx     = [10, 16, 23, 37, 45]
y[idx] .= 5

X = x.^[0 1 2 3 4]
A = [X -I; -X -I]
b = [y; -y]
c = [zeros(5);ones(N)]

θ = linprog_Convex(c,A,(<=),b)

t    = range(-1,stop=1,length=200)
yhat = (t.^[0 1 2 3 4])*θ[1:5]

p5 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft)
plot!(t,yhat,linecolor=:red,linewidth=4,label="fitted curve")
display(p5)
##------------------------------------------------------------------------------

# Ch 7.2 Overfitting
# Julia: An overfitting example

using LegendrePolynomials

N = 20
x = sort(rand(N)*2 .- 1)
a = [-0.001, 0.01, 0.55, 1.5, 1.2]
y = hcat([Pl.(x,p) for p=0:4]...)*a + 0.1*randn(N)

P = 20
X = hcat([Pl.(x,p) for p=0:P]...)

β = X\y

t    = range(-1,stop=1,length=50)
Xhat = hcat([Pl.(t,p) for p=0:P]...)
yhat = Xhat*β

p6 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft,ylims = (-3,3))
plot!(t,yhat,linecolor=:blue,linewidth=4,label="fitted curve")
display(p6)
##------------------------------------------------------------------------------

using Statistics

Nset = round.( Int,exp10.(range(1,stop=3,length=20)) )  #10,13,16,21,...

E_train = zeros(length(Nset))
E_test  = zeros(length(Nset))
a = [1.3, 2.5]
for j = 1:length(Nset)
    local N,x,E_train_temp,E_test_temp,X
    N = Nset[j]
    x = range(-1,stop=1,length=N)
    E_train_temp = zeros(1000)
    E_test_temp  = zeros(1000)
    X = [ones(N) x]
    for i = 1:1000
        local y,y1,θ,yhat
        y  = a[1] .+ a[2]*x + randn(N)
        y1 = a[1] .+ a[2]*x + randn(N)
        θ = X\y
        yhat = θ[1] .+ θ[2]*x
        E_train_temp[i] = mean(abs2,yhat-y)
        E_test_temp[i]  = mean(abs2,yhat-y1)
    end
    E_train[j] = mean(E_train_temp)
    E_test[j]  = mean(E_test_temp)
end

p7 = scatter(Nset,E_train,xscale=:log10,markershape=:x,markercolor=:black,label="Training Error")
scatter!(Nset,E_test,xscale=:log10,markershape=:circle,markercolor=:red,label="Testing Error")
plot!(Nset,1.0 .- 2.0./Nset,xscale=:log10,linestyle=:solid,color=:black,label="")
plot!(Nset,1.0 .+ 2.0./Nset,xscale=:log10,linestyle=:solid,color=:red,label="")
display(p7)
##------------------------------------------------------------------------------

# Ch 7.3 Bias and Variance
# Julia code to visualize the average predictor

N = 20
a = [5.7, 3.7, -3.6, -2.3, 0.05]
x = range(-1,stop=1,length=N)
X = x.^[0 1 2 3 4]

t    = range(-1,stop=1,length=50)
tMat = t.^[0 1 2 3 4]

yhat = zeros(50,100)                      #50x100 instead of 100x50
for i = 1:100
    local y,θ
    y = X*a + 0.5*randn(N)
    θ = X\y
    yhat[:,i] = tMat*θ
end

p8 = plot(t,yhat,linecolor=:gray,label="")
plot!(t,mean(yhat,dims=2),linecolor=:red,linewidth=4,label="")
display(p8)
##------------------------------------------------------------------------------

# Ch 7.4 Regularization
#  Julia code to demonstrate a ridge regression example

#----------------------------------------------------------
"""
    RidgeRegression(Y,X,λ,β₀=0)

Calculate ridge regression estimate with targetr vector β₀.
"""
function RidgeRegression(Y,X,λ,β₀=0)
    K = size(X,2)
    isa(β₀,Number) && (β₀=fill(β₀,K))
    b = (X'X+λ*I)\(X'Y+λ*β₀)      #same as inv(X'X+λ*I)*(X'Y+λ*β₀)
    return b
end
#----------------------------------------------------------

N = 20                                 # Generate data
x = range(-1,stop=1,length=N)
a = [0.5, -2, -3, 4, 6]
y = x.^[0 1 2 3 4]*a + 0.25*randn(N)

λ = 0.1                                # Ridge regression
d = 20
X = x.^collect(0:d-1)'
A = [X; sqrt(λ)I]
b = [y; zeros(d)]
θ = A\b
θ_alt = RidgeRegression(y,X,λ)         # alternative ridge regression
display([θ θ_alt])

t    = range(-1,stop=1,length=500)
Xhat = t.^collect(0:d-1)'
yhat = Xhat*θ

p9 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft)
plot!(t,yhat,linecolor=:blue,linewidth=4,label="fitted curve")
display(p9)
##------------------------------------------------------------------------------

#----------------------------------------------------------
"""
    LassoEN(Y,X,λ,δ,β₀)

Do Lasso (set λ>0,δ=0), ridge (set λ=0,δ>0) or elastic net regression (set λ>0,δ>0).
Setting β₀ allows us to specify the target level.

"""
function LassoEN(Y,X,λ,δ=0.0,β₀=0)

    K = size(X,2)
    isa(β₀,Number) && (β₀=fill(β₀,K))

    #b_ls = X\Y                    #LS estimate of weights, no restrictions

    Q  = X'X
    c  = X'Y                      #c'b = Y'X*b

    b  = Variable(K)              #define variables to optimize over
    L1 = quadform(b,Q)            #b'Q*b
    L2 = dot(c,b)                 #c'b
    L3 = norm(b-β₀,1)             #sum(|b-β₀|)
    L4 = sumsquares(b-β₀)         #sum((b-β₀)^2)

    Sol = minimize(L1-2*L2+λ*L3+δ*L4)      #u'u + λ*sum(|b|) + δ*sum(b^2), where u = Y-Xb
    solve!(Sol,()->SCS.Optimizer(verbose = false))
    b_i = vec(evaluate(b))

    return b_i

end
#----------------------------------------------------------

using LinearAlgebra, DelimitedFiles, Convex, SCS
import MathOptInterface
const MOI = MathOptInterface

data  = readdlm("dataset/ch7_data_crime.txt")
y     = data[:,1]        # the observed crime rate
X     = data[:,3:end]    # feature vectors
(N,d) = size(X)

(y,X) = (y.-mean(y),X.-mean(X,dims=1))           # de-meaning all variables

λset = exp10.(range(-2,stop=4,length=50))        # adjusted range of λ values since X/100 is used
(θ_store,θ_store2,θ_store3) = [zeros(50,d) for _=1:3]   # 50 x d instead of d x 50
for i = 1:length(λset)
    θ_store[i,:]  = LassoEN(y/100,X/100,λset[i])    # LASSO, /100 improves the numerical stability
    θ_store2[i,:] = LassoEN(y/100,X/100,0,λset[i])  # ridge regression
    #θ_store3[i,:] = RidgeRegression(y/100,X/100,λset[i])   # alternative ridge regression
end
#display(θ_store2-θ_store3)                                 # compare solutions

p10 = plot(λset,θ_store,xscale=:log10,title="LASSO",xlabel="λ",ylabel="feature attribute",
           label = ["funding" "% high" "% no high" "% college" "% graduate"],
            linecolor = [:blue :red :yellow3 :magenta :green],ylims=(-5,15))
display(p10)

p10b = plot(λset,θ_store2,xscale=:log10,title="ridge regression",xlabel="λ",ylabel="feature attribute",
           label = ["funding" "% high" "% no high" "% college" "% graduate"],
            linecolor = [:blue :red :yellow3 :magenta :green],ylims=(-5,15))
display(p10b)
##------------------------------------------------------------------------------

# Julia code to demonstrate overfitting and LASSO

using LinearAlgebra, LegendrePolynomials, Convex, SCS
import MathOptInterface
const MOI = MathOptInterface

N = 20                                          # Generate data
x = range(-1,stop=1,length=N)
a = [1, 0.5, 0.5, 1.5, 1]
y = hcat([Pl.(x,p) for p=0:4]...)*a + 0.25*randn(N)

d = 20                                          # Solve LASSO using Convex.jl
X = hcat([Pl.(x,p) for p=0:d-1]...)
λ = 2
θ = LassoEN(y,X,λ)

t    = range(-1,stop=1,length=200)
Xhat = hcat([Pl.(t,p) for p=0:d-1]...)
yhat = Xhat*θ

p11 = scatter(x,y,makershape=:circle,label="data",legend=:bottomleft)
plot!(t,yhat,linecolor=:blue,linewidth=4,label="fitted curve")
display(p11)
#------------------------------------------------------------------------------


"""
	fourier_matrix(n::Int; normalized=true)

Returns a ``n``-by-``n`` Fourier matrix with optional normalization (`true` by default).
"""
function fourier_matrix(n::Int; normalized = true)

    U = Array{ComplexF64}(undef, n, n)

    for i in 1:n
        for j in 1:n
            U[i,j] = exp((2im * pi / n)* (i-1) * (j-1))
        end
    end

    if normalized
        return 1/sqrt(n) * U
    else
        return U
    end
end

"""
	hadamard_matrix(n::Int; normalized=true)

Returns a ``n``-by-``n`` Hadamard matrix with optional normalization (`true` by default).
"""
function hadamard_matrix(n::Int, normalized = true )

	H = Array{ComplexF64}(undef, n, n)

	for i in 1:n
		for j in 1:n
			bi = map(x->parse(Int,x), split(bitstring(Int8(i-1)),""))
			bj = map(x->parse(Int,x), split(bitstring(UInt8(j-1)),""))
			H[i,j] = (-1)^(sum(bi.&bj))
		end
	end

	if normalized
		return 1/sqrt(n) * H
	else
		return H
	end

end

"""
	sylvester_matrix(p::Int; normalized=true)

Returns a ``2^p``-by-``2^p`` Sylvester matrix with optional normalization
(`true` by default) following [https://arxiv.org/abs/1502.06372](https://arxiv.org/abs/1502.06372).
"""
function sylvester_matrix(p; normalized = true)

    """returns the sylvester matrix of size 2^p"""

    function sylvester_element(i,j,p)

        """returns the element A(i,j) for the sylvester matrix of size 2^p"""
        # following https://arxiv.org/abs/1502.06372
        if p == 0
            return 1
        else
            i_b = digits(i-1, base = 2, pad = p+1)
            j_b = digits(j-1, base = 2, pad = p+1)
            return (-1)^(sum(i_b .* j_b))
        end
    end

    A = Array{ComplexF64}(undef, 2^p, 2^p)
    for i in 1:2^p
        for j in 1:2^p
            A[i,j] = sylvester_element(i,j,p)
        end
    end

    normalized == true ? 1/sqrt(2^p) * A : A

end

"""
	rand_haar(n::Int)

Returns a ``n``-by-``n`` Haar distributed unitary matrix following [https://case.edu/artsci/math/esmeckes/Meckes_SAMSI_Lecture2.pdf](https://case.edu/artsci/math/esmeckes/Meckes_SAMSI_Lecture2.pdf).
"""
function rand_haar(n::Int)

    """generates a Haar distributed unitary matrix of size n*n"""
    # follows https://case.edu/artsci/math/esmeckes/Meckes_SAMSI_Lecture2.pdf

    qr(randn(ComplexF64, n,n)).Q
end

function matrix_test(n::Int)

    """matrix of 1,2,3... for testing your code"""

    U = Array{Int64}(undef, n, n)

    for i in 1:n
        for j in 1:n
            U[i,j] = (i-1) * n + j
        end
    end

    U

end

function antihermitian_test_matrix(n::Int)

    h = randn(ComplexF64, n, n)

    for i = 1:n
        h[i,i] = 1im*imag(h[i,i])
    end

    for i = 1:n
        for j = i+1:n
            h[j,i] = -conj(h[i,j])
        end
    end

    h

end

function hermitian_test_matrix(n::Int)

    h = randn(ComplexF64, n,n)

    for i = 1:n
        h[i,i] = real(h[i,i])
    end

    for i = 1:n
        for j = i+1:n
            h[j,i] = conj(h[i,j])
        end
    end

    h

end

function permutation_matrix_special_partition_fourier(n::Int)

    """P * [1,1,1,1,1,0,0,0,0,0] = [1, 0, 1, 0, 1, 0, 1, 0, 1, 0]"""

    P = zeros(Int, n,n)

    for i in 1:2:n
        P[i, Int((i+1)/2)] = 1
    end

    for i in 2:2:n
        P[i, Int(n/2 + i/2)] = 1
    end

    P

end

"""
	rand_gram_matrix_from_orthonormal_basis(n::Int, r::Int)

Returns a ``n``-by-``n`` random Gram matrix that generates orthonormal basis of `r` vectors.
"""
function rand_gram_matrix_from_orthonormal_basis(n,r)

	"""generates a random gram matrix with the property that the
	generating vector basis of r vectors is orthonormal (as is
	strangely the case in Drury's work)"""

	function normalized_random_vector(n)
		v = rand(ComplexF64, n)
		1/norm(v) .* v
	end

	if r >= n
		throw(ArgumentError("need rank < dim to have a non trivial result"))
	end

	generating_vectors = 0.
	#while det(generating_vectors) ≈ 0. #check it's a basis
	    generating_vectors = hcat([normalized_random_vector(n) for i = 1:r]...)
	#### TODO caveat : we don't check for linear independance so there is a chance it doesn't work though highly unlikely
	#end

	generating_vectors = modified_gram_schmidt(generating_vectors)

	is_orthonormal(generating_vectors)

	generating_vectors * generating_vectors'

end

function modified_gram_schmidt(input_vectors)

	"""does the modified gram schmidt (numerically stable) on the columns of the matrix input_vectors"""

	function projector(a, u)
		"""projector of a on u"""
		dot(u,a)/dot(u,u) .* u
	end

	final_vectors = copy(input_vectors)

	for i in 1:size(input_vectors)[2]

		if i == 1
			#normalize first vector
			final_vectors[:,1] /= norm(final_vectors[:, 1])
		else
			for j in 1:i-1
				final_vectors[:, i] -= projector(final_vectors[:, i], final_vectors[:, j])
			end
			final_vectors[:, i] /= norm(final_vectors[:, i])

		end

	end

	is_orthonormal(final_vectors)

	final_vectors

end

"""
	rand_gram_matrix(n::Int)

Returns a full rank ``n``-by-``n`` random Gram matrix.
"""
function rand_gram_matrix(n::Int)

	"""random gram matrix with full rank """

	function normalized_random_vector(n)
		v = rand(ComplexF64, n)
		1/norm(v) .* v
	end

	generating_vectors = 0.
	while det(generating_vectors) ≈ 0. #check it's a basis
	    generating_vectors = hcat([normalized_random_vector(n) for i = 1:n]...)

	end

	generating_vectors' * generating_vectors

end

"""
	rand_gram_matrix_rank(n::Int, r::Int)

Returns a ``n``-by-``n`` random Gram matrix of maximum rank and great likelihood `r`.
"""
function rand_gram_matrix_rank(n, r)

	"""random gram matrix with rank at most r, and with great likelihood r
	"""

	function normalized_random_vector(r)
		v = rand(ComplexF64, r)
		1/norm(v) .* v
	end

	generating_vectors = 0.
	generating_vectors = hcat([normalized_random_vector(r) for i = 1:n]...)

	generating_vectors' * generating_vectors

end

"""
	rand_gram_matrix_real(n::Int)

Returns a real ``n``-by-``n`` random Gram matrix of full rank.
"""
function rand_gram_matrix_real(n)

	"""random gram matrix with full rank and real el"""

	function normalized_random_vector(n)
		v = rand(Float64, n)
		1/norm(v) .* v
	end

	generating_vectors = 0.
	while det(generating_vectors) ≈ 0. #check it's a basis
	    generating_vectors = hcat([normalized_random_vector(n) for i = 1:n]...)

	end

	generating_vectors' * generating_vectors

end

"""
	rand_gram_matrix_positive(n::Int)

Returns a positive elements ``n``-by-``n`` random Gram matrix of full rank.
"""
function rand_gram_matrix_positive(n::Int)

	"""random gram matrix with full rank and positive el"""

	function normalized_random_vector(n)
		v = abs.(rand(Float64, n))
		1/norm(v) .* v
	end

	generating_vectors = 0.
	while det(generating_vectors) ≈ 0. #check it's a basis
	    generating_vectors = hcat([normalized_random_vector(n) for i = 1:n]...)

	end

	generating_vectors' * generating_vectors

end

# functions to clear matrices for better readability of a matrix by a human

function set_small_values_to_zero(x, eps)
	abs(x) < eps ? 0.0 : x
end

function set_small_imag_real_to_zero(x, eps = 1e-7)
	if abs(real(x)) < eps
		x = 0.0 + 1im * imag(x)
	end
	if abs(imag(x)) < eps
		x = real(x) + 0.0im
	end
	x
end

function make_matrix_more_readable(M)
	set_small_imag_real_to_zero.(M)
end

function display_matrix_more_readable(M)
	pretty_table(make_matrix_more_readable(M))
end

####### sub matrices ########

function remove_row_col(A, rows_to_remove, cols_to_remove)

	"""hands back the matrix A without the rows labelled in rows_to_remove, columns in cols_to_remove

	ex :
	rows_to_remove = [1]
	cols_to_remove = [2]

	to have A(1,2) in the notations of Minc"""

	n = size(A)[1]
	m = size(A)[2]

	range_rows = collect(1:n)
	range_col = collect(1:m)

	setdiff!(range_rows, rows_to_remove)
	setdiff!(range_col, cols_to_remove)

	A[range_rows, range_col]

end

### permanent related quantities ###

function sub_permanent_matrix(A)

	n = size(A)[1]

	sub_permanents = similar(A)

	for i in 1 : n
		for j in 1 : n

			sub_permanents[i,j] = ryser(remove_row_col(A, [i], [j]))

		end
	end

	sub_permanents

end

function gram_from_n_r_vectors(M)

	# generate a random matrix of n r-vectors M
	# normalize it to have a gram matrix

	n = size(M)[1]

	for i in  1 : n
		M[:,i] ./= norm(M[:,i])
	end

	M' * M # our rank r n*n gram matrix

end

"""
	gram_matrix_one_param(n::Int, x::Real)

Returns a ``n``-by-``n`` Gram matrix parametrized by the real ``0 ≤ x ≦ 1``.
"""
function gram_matrix_one_param(n::Int, x::Real)

	@argcheck x>=0 && x<= 1

	S = 1.0 * Matrix(I, n, n)
	for i in 1:n
		for j in 1:n
			i!=j ? S[i,j] = x : continue
		end
	end

	S

end

function column_normalize(M)
   for col in 1:size(M,2) #column normalize
	   M[:, col] = M[:, col]./norm(M[:, col])
   end
   M
end

"""
	perturbed_gram_matrix(M, epsilon)

Returns a Gram matrix generated by the columns of `M` which are perturbed by
a Gaussian quantity of variance epsilon once normalized.
"""
function perturbed_gram_matrix(M, epsilon)

    """M defines the set of vectors generating the gram matrix
    each column is a generating vector for the gram matrix
    we perturb them by some random gaussian amount with set variance epsilon once normalized """

    M = column_normalize(M)

    d = Normal(0.0, epsilon)

    perturbation_vector = rand(d,size(M))

    M += perturbation_vector

    M = column_normalize(M)

    M' * M

end

"""
	perturbed_unitary(U, epsilon)

Returns a unitary matrix whose columns are generating vector perturbed by a
random Gaussian quantity with variance epsilon once normalized.
"""
function perturbed_unitary(U, epsilon)

    """U a unitary matrix each column is a generating vector
    we perturb them by some random gaussian amount with set variance epsilon once normalized """

    d = Normal(0.0, epsilon)

    perturbation_vector = rand(d,size(U))

    U += perturbation_vector

    U = modified_gram_schmidt(U)

    U

end


"""
# Counting Sort
OVERVIEW:
Counting Sort is a sorting algorithm that sort elements within a specific range.
The sorting technique is to count the existing element and stored its occurrence time in a new list, then only print it out.

STEPS:
Assume the input as --> x=[-3, 1, -5, 0, -3]
  minimum = -5
- STEP 1: Create a list size within the range, in this case is -5 --> 1 which have range of 7 (-5, -4, -3, -2, -1, 0, 1), so list with size 7 and assign all to 0 is created
- STEP 2: Count the occurances of element in the list
          First number = -3 it is the third number in the range, so count[3]+=1
          Final view:
          index : ( 1,  2,  3,  4,  5, 6, 7)
          range : (-5, -4, -3, -2, -1, 0, 1)
          count : [ 1,  0,  2,  0,  0, 1, 1] <-- the list will store this occurences
- STEP 3: Make the count list accumulate the occurances
          The final count is (1, 1, 3, 3, 3, 4, 5)
- STEP 4: Assign the elements in x into correct possition by creating a new list (will call 'output' in this sample)
          the 1st element in 'x' is -3, it is third in range, so it will call the index of 3 in 'count', which is 3 and assign the -3 in to 3rd position in 'output',
          then the third element in range will deduct by 1, so the next repeated element will get the correct position, new 'count' --> [1, 1, 2, 3, 3, 4, 5]

          the 2nd element in 'x' is  1, it is last  in range, so it will call the index of 7 in 'count', which is 5 and assign the  1 in to 5th position in 'output',
          new 'count' --> [1, 1, 2, 3, 3, 4, 4]
          ......
          ......
          *If you want the order of original array to have the same order as the output array use can change this to decremental for loop
- STEP 5: Assign the 'ouput' list back to 'x'

FINAL RESULT -->  [-5, -3, -3, 0, 1]
"""
function counting_sort!(
    arr::Vector{T},
    l::Int = 1,
    r::Int = length(arr),
) where {T}
    if l >= r
        return nothing
    end
    max = maximum(arr)
    min = minimum(arr)
    range = max - min + 1
    count = Vector{T}(undef, range)
    output = Vector{T}(undef, r)

    for i in 1:range
        count[i] = 0
    end

    # Store count of the item that appear in the 'arr' (STEP 2)
    for i in 1:r
        count[arr[i]-min+1] += 1
    end

    # Calculate cumulative frequency in 'count' (STEP 3)
    for i in 2:length(count)
        count[i] += count[i-1]
    end

    # Build the 'output' by assigning the item into correct position (STEP 4)
    for i in 1:r
        index = arr[i] - min + 1
        output[count[index]] = arr[i]
        count[index] -= 1
    end

    # Copy the 'output' to 'arr', so that 'arr' contain sorted item (STEP 5)
    for i in 1:r
        arr[i] = output[i]
    end
end


"""
    perfect_cube(number)

Check if a number is a perfect cube or not.

# Example
```jula
perfect_cube(27) # returns true
perfect_cube(4)  # returns false
```

Contributed By:- [Ashwani Rathee](https://github.com/ashwani-rathee) and [Rratic](https://github.com/Rratic)
"""
function perfect_cube(number::N)where N<:Integer
    val = trunc(cbrt(number))
    return (val * val * val) == number
end


"""
    opnorm(A::IntervalMatrix, p::Real=Inf)

The matrix norm of an interval matrix.

### Input

- `A` -- interval matrix
- `p` -- (optional, default: `Inf`) the class of `p`-norm

### Notes

The matrix ``p``-norm of an interval matrix ``A`` is defined as

```math
    ‖A‖_p := ‖\\max(|\\text{inf}(A)|, |\\text{sup}(A)|)‖_p
```

where ``\\max`` and ``|·|`` are taken elementwise.
"""
function LinearAlgebra.opnorm(A::IntervalMatrix, p::Real=Inf)
    if p == Inf
        return _opnorm_inf(A)
    elseif p == 1
        return _opnorm_1(A)
    else
        error("the interval matrix norm for this value of p=$p is not implemented")
    end
end

# The operator norm in the infinity norm corresponds to the
# maximum absolute value row-sum.
function _opnorm_inf(A::IntervalMatrix{T}) where {T}
    m, n = size(A)
    res = zero(T)
    @inbounds @simd for i in 1:m
        acc = zero(T)
        for j in 1:n
            x = A[i, j]
            acc += max(abs(inf(x)), abs(sup(x)))
        end
        if acc > res
            res = acc
        end
    end
    return res
end

# The operator norm in the 1-norm corresponds to the
# maximum absolute value column-sum.
function _opnorm_1(A::IntervalMatrix{T}) where {T}
    m, n = size(A)
    res = zero(T)
    @inbounds @simd for j in 1:n
        acc = zero(T)
        for i in 1:m
            x = A[i, j]
            acc += max(abs(inf(x)), abs(sup(x)))
        end
        if acc > res
            res = acc
        end
    end
    return res
end

"""
    diam_norm(A::IntervalMatrix, p=Inf)

Return the diameter norm of the interval matrix.

### Input

- `A` -- interval matrix
- `p` -- (optional, default: `Inf`) the `p`-norm used; valid options are:
         `1`, `2`, `Inf`

### Output

The operator norm, in the `p`-norm, of the scalar matrix obtained
by taking the element-wise `diam` function, where `diam(x) := sup(x) - inf(x)`
for an interval `x`.

### Notes

This function gives a measure of the *width* of the interval matrix.
"""
function diam_norm(A::IntervalMatrix, p=Inf)
    return opnorm(diam.(A), p)
end


# # Convex hull of a set of points

# ## Convex hull in a plane

# This examples shows how to find the convex hull in the context of a plane.
# First we have to create an object representing the vertices. We have multiple ways of doing this

using Polyhedra

v = convexhull([0, 0], [0, 1], [1, 0], [0.1, 0.1]) # list of points
v = vrep([[0, 0], [0, 1], [1, 0], [0.1, 0.1]]) # vector of points
# number of points × dimension matrix
x = [0,0,1,0.1]
y = [0,1,0,0.1]
v = vrep([x y])

# Then we can compute the hull of these points using the planar_hull function

Polyhedra.planar_hull(v)

# ## Convex hull in higher dimension

# In higher dimension, we can do it with a linear programming solver implementing the [MathOptInterface](https://github.com/jump-dev/MathOptInterface.jl), e.g.,

import GLPK

removevredundancy(v, GLPK.Optimizer)

# We can also use any Polyhedral library implementing the interface of this package. If we don't specify any library, it falls back to a default one implementing on this package which will use the `planar_hull` if the dimension is 2 (so it's equivalent to the first approach shown above):

p = polyhedron(v)
removevredundancy!(p)
p

# We can also specify a library. For instance, to use `CDDLib`, write `using CDDLib` and then `p = polyhedron(v, CDDLib.Library())`.

examples/Convex hull of a set of points.jl


# Huber loss function

using LinearAlgebra

export HuberLoss

"""
**Huber loss**

    HuberLoss(ρ=1.0, μ=1.0)

Returns the function
```math
f(x) = \\begin{cases}
    \\tfrac{μ}{2}\\|x\\|^2 & \\text{if}\\ \\|x\\| ⩽ ρ \\\\
    ρμ(\\|x\\| - \\tfrac{ρ}{2}) & \\text{otherwise},
\\end{cases}
```
where `ρ` and `μ` are positive parameters.
"""
struct HuberLoss{R <: Real} <: ProximableFunction
    rho::R
    mu::R
    function HuberLoss{R}(rho::R, mu::R) where {R <: Real}
        if rho <= 0.0 || mu <= 0.0
            error("parameters rho and mu must be positive")
        else
            new(rho, mu)
        end
    end
end

is_convex(f::HuberLoss) = true
is_smooth(f::HuberLoss) = true

HuberLoss(rho::R=1.0, mu::R=1.0) where {R <: Real} = HuberLoss{R}(rho, mu)

function (f::HuberLoss)(x::AbstractArray{T}) where T <: Union{Real, Complex}
    normx = norm(x)
    if normx <= f.rho
        return (f.mu/2)*normx^2
    else
        return f.rho*f.mu*(normx-f.rho/2)
    end
end

function gradient!(y::AbstractArray{T}, f::HuberLoss, x::AbstractArray{T}) where T <: Union{Real, Complex}
    normx = norm(x)
    if normx <= f.rho
        y .= f.mu .* x
        v = (f.mu/2)*normx^2
    else
        y .= (f.mu*f.rho)/normx .* x
        v = f.rho*f.mu*(normx-f.rho/2)
    end
    return v
end

function prox!(y::AbstractArray{T}, f::HuberLoss, x::AbstractArray{T}, gamma::Real=1.0) where T <: Union{Real, Complex}
    normx = norm(x)
    mugam = f.mu*gamma
    scal = (1-min(mugam/(1+mugam), mugam*f.rho/(normx)))
    for k in eachindex(y)
        y[k] = scal*x[k]
    end
    normy = scal*normx
    if normy <= f.rho
        return (f.mu/2)*normy^2
    else
        return f.rho*f.mu*(normy-f.rho/2)
    end
end

fun_name(f::HuberLoss) = "Huber loss"
fun_dom(f::HuberLoss) = "AbstractArray{Real}, AbstractArray{Complex}"
fun_expr(f::HuberLoss) = "x ↦ (μ/2)||x||² if ||x||⩽ρ, μρ(||x||-ρ/2) otherwise"
fun_params(f::HuberLoss) = string("ρ = $(f.rho), μ = $(f.mu)")

function prox_naive(f::HuberLoss, x::AbstractArray{T}, gamma::Real=1.0) where T <: Union{Real, Complex}
    y = (1-min(f.mu*gamma/(1+f.mu*gamma), f.mu*gamma*f.rho/(norm(x))))*x
    if norm(y) <= f.rho
        return y, (f.mu/2)*norm(y)^2
    else
        return y, f.rho*f.mu*(norm(y)-f.rho/2)
    end
end


# This subroutine discretize an AR(1) process with Tauchen (86)
# λ' = (1 - ρ)μ_λ + ρλ + ϵ, where ϵ ~ N(0, (σ_ϵ)^2)
# Inputs:
# Outputs:

using Distributions

function Tauchen(μ_λ, ρ, σ_ϵ, n; m = 3)
    σ_λ = σ_ϵ/(1-ρ^2)^(1/2);

    λ_min = μ_λ - m*σ_λ;
    λ_max = μ_λ + m*σ_λ;
    λ_vals = range(λ_min, λ_max, length = n); # can also use LinRange(λ_min, λ_max, n)

    w = λ_vals[2] - λ_vals[1];

    d = Normal();
    P = zeros(n,n);

    for i in 1:n
        for j = 1:n
            if j == 1
                P[i,j] = cdf(d, (λ_vals[1] + w/2 - (1-ρ)*μ_λ - ρ*λ_vals[i])/σ_ϵ);
            elseif j == n
                P[i,j] = 1 - cdf(d, (λ_vals[n] - w/2 - (1-ρ)*μ_λ - ρ*λ_vals[i])/σ_ϵ);
            else
                P[i,j] = cdf(d, (λ_vals[j] + w/2 - (1-ρ)*μ_λ - ρ*λ_vals[i])/σ_ϵ) - cdf(d, (λ_vals[j] - w/2 - (1-ρ)*μ_λ - ρ*λ_vals[i])/σ_ϵ);
            end

        end
    end
    return λ_vals, P
end


# # Logistic Regression
# The presented example is adapted from the [Machine Learning - course by Andrew Ng](https://www.coursera.org/learn/machine-learning).
#
# In this example we use logistic regression to estimate the parameters $\theta_i$ of a logistic model in a classification problem. We will first  transform the logistic regression problem into an exponential cone optimisation problem. We will then solve the optimisation problem with COSMO  and determine the model parameters and the decision boundary.
# ## Visualizing the data
# Before we start, let's load and take a look at the example data from `examples/chip_data.txt`:

#-
using LinearAlgebra, SparseArrays, COSMO, JuMP
using Plots

## load example data
f = open(joinpath(@__DIR__, "chip_data.txt"))
lines = readlines(f)
close(f)
n_data = length(lines)
n_half = div(n_data, 2)
x1 = zeros(n_data)
x2 = zeros(n_data)
y = zeros(Float64, n_data)
for (i, l) in enumerate(lines)
    s = split(l, ",")
    x1[i] = parse(Float64, s[1])
    x2[i] = parse(Float64, s[2])
    y[i] = parse(Float64, s[3])
end

## visualize data
plot(x1[1:n_half-1], x2[1:n_half-1], color = :blue, st=:scatter, markershape = :cross, aspect_ratio=:equal, label = "Accepted", xlabel = "x1 - Microchip Test Score 1", ylabel = "x2 - Microchip Test Score 2")
plot!(x1[n_half:end], x2[n_half:end], color = :red, st=:scatter, markershape = :circle, label = "Rejected")

# The plot shows two test scores of $n$ microchip samples from a fabrication plant and whether the chip passed the quality check. Based on this data we would like to build a logistic model that takes into account the test scores and helps us predict the likelihood of a chip being accepted.
#
# ## Defining the logistic model
# The logistic regression hypothesis is given by
#
# $$
# h_\theta(x) = g(\theta^\top x)
# $$
# where $g$ is the sigmoid function:
# $$
# g(\theta^\top x) = \frac{1}{1+\exp(-\theta^\top x)}.
# $$
#
# The vector $x$ represents the independent variables and $\theta$ represents the model parameters. For our samples we set the dependent variable $y =1$ if the chip was accepted and $y = 0$ otherwise.
#
# The function $h_\theta(x)$ can be interpreted as the probability of the outcome being true rather than false. We want to find the parameters $\theta$ that maximize the log-likelihood over all (independently Bernoulli distributed) observations
#
# $$
# J(\theta) = \sum_{i, y_i = 1} \log h_\theta(x_i) + \sum_{i, y_i = 0} \log (1-h_\theta(x_i)).
# $$
# Consequently, we want to solve the following optimization problem:
# $$
# \text{minimize} \quad -J(\theta) + \mu \|\theta \|_2,
# $$
#
# where we added a regularization term with parameter $\mu$ to prevent overfitting.

# ## Feature mapping
# As our dataset only has two independent variables (the test scores) our model $y = \theta_0 + \theta_1 x_1 + \theta_2 x_2$ will have the form of a straight line. Looking at the plot one can see that a line will not perform well in separating the samples. Therefore, we will create more features based on each data point by mapping the original features ($x_1$, $x_2$) into all polynomial terms of $x_1$ and $x_2$ up to the 6th power:
#
# $$
# \text{map\_feature}(x_1,x_2) = [1, x_1, x_2, x_1^2, x_1x_2, x_2^2, x_1^3, \dots, x_1x_2^5, x_2^6 ]
# $$
#
# This will create 28 features for each sample.

#-

function map_feature(x1, x2)
  deg = 6
  x_new = ones(length(x1))
  for i = 1:deg, j = 0:i
      x_new = hcat(x_new, x1.^(i-j) .* x2.^j)
  end
  return x_new
end

X = map_feature(x1, x2);
size(X)

# ## Transformation into a conic optimisation problem
# We can rewrite above likelihood maximisation problem as a conic optimisation problem with exponential-cone-, second-order-cone-, equality-, and inequality constraints:
#
# $$
# \begin{array}{ll}
# \text{minimize}  &\sum_i^n \epsilon_i + \mu v\\
# \text{subject to}  & \|\theta \|_2 \leq v\\
# &  \log(1 + \exp(-\theta^\top x_i)) \leq \epsilon_i  \quad \text{if } y_i = 1, \\
# & \log(1 + \exp(\theta^\top x_i)) \leq \epsilon_i  \quad\text{   otherwise.}
# \end{array}
# $$
#
# Implementing the constraint $\log(1 + \exp(z)) \leq \epsilon$ for each of the $n$ samples requires two exponential cone constraints, one inequality constraint and two equality constraints. To see this, take the exponential on both sides and then divide by $\exp(\epsilon)$ to get:
#
# $$
# \exp(-\epsilon) + \exp(z - \epsilon) \leq 1.
# $$
#
# This constraint is equivalent to:
# $$
# \begin{array}{ll}
# (z - \epsilon, s_1, t_1) &\in K_{\text{exp}}, \\
# (-\epsilon, s_2, t_2) &\in K_{\text{exp}}, \\
# t_1 + t_2 &\leq 1,\\
# s_1 = s_2 &= 1,
# \end{array}
# $$
#
# where we defined the exponential cone as:
# $$
# K_{\text{exp}} = \{(r, s, t) \mid s >0, s \exp(r/s) \leq t \} \cup \{ r \leq 0, s = 0, t \geq 0 \}.
# $$
#
# Based on this transformation our optimisation problem will have $5n + n_\theta + 1$ variables, 1 SOCP constraint, $2n$ exponential cone
# constraints, $n$ inequality constraints and $2n$ equality constraints.
# Let's model the problem with JuMP and COSMO:

#-

n_theta = size(X, 2)
n = n_data
μ  = 1.

m = JuMP.Model(COSMO.Optimizer)
@variable(m, v)
@variable(m, θ[1:n_theta])
@variable(m, e[1:n])
@variable(m, t1[1:n])
@variable(m, t2[1:n])
@variable(m, s1[1:n])
@variable(m, s2[1:n])

@objective(m, Min, μ * v + sum(e))
@constraint(m, [v; θ] in MOI.SecondOrderCone(n_theta + 1))

## create the constraints for each sample points
for i = 1:n
  yi = y[i]
  x = X[i, :]
  yi == 1. ? (a = -1) : (a = 1)
  @constraint(m, [a * dot(θ, x) - e[i]; s1[i]; t1[i] ] in MOI.ExponentialCone())
  @constraint(m, [-e[i]; s2[i]; t2[i]] in MOI.ExponentialCone())
  @constraint(m, t1[i] + t2[i] <= 1)
  @constraint(m, s1[i] == 1)
  @constraint(m, s2[i] == 1)
end
JuMP.optimize!(m)
#-
theta = value.(θ)

# Once we have solved the optimisation problem and obtained the parameter vector $\theta$, we can plot the decision boundary. This can be done by evaluating our model over a grid of points $(u,v)$ and then plotting the contour line where the function $g$ returns a probability of $p=0.5$.

#-
## First we evaluate our model over a grid of points z = θ' x
u = collect(range(-1., stop = 1.5, length = 50))
v = collect(range(-1., stop = 1.5, length = 50))
z = zeros(length(u), length(v));
for i = 1:length(u), j = 1:length(v)
    z[i, j] = dot(map_feature(u[i], v[j]), theta);
end

# To add the decision boundary we have to plot the line indicating $50\%$ probability of acceptance, i.e. $g(\theta^\top x) = g(z)  = 0.5$ which we get at $z=0$.

#-
plot(x1[1:n_half-1], x2[1:n_half-1], color = :blue, st = :scatter, markershape = :cross, aspect_ratio=:equal, label = "Accepted", xlabel = "x1 - Microchip Test Score 1", ylabel = "x2 - Microchip Test Score 2")
plot!(x1[n_half:end], x2[n_half:end], color = :red, st = :scatter, markershape = :circle, label = "Rejected")
contour!(u, v, z', levels = [0.], c = :black, linewidth = 2)

# ## Solving the optimisation problem directly with COSMO
# We can solve the problem directly in COSMO by using its modeling interface. The problem will have $nn = 5 n + n_\theta + 1$ variables. Let us define the cost function  $\frac{1}{2}x^\top P x + q^\top x$:

#-
nn = 5 * n + n_theta +  1
P = spzeros(nn, nn)
q = zeros(nn)
q[1] = μ
for i = 1:n
  q[1 + n_theta + (i - 1) * 5 + 1] = 1.
end
# Next we define a function that creates the `COSMO.Constraints` for a given sample:

#-
## the order of the variables
## v, thetas, [e1 t11 t12 s11 s12] [e2 t21 t22 s21 s22] ...
## for each sample create two exponential cone constraints,
## 1 nonnegatives constraint, 2 zeroset constraints
function add_log_regression_constraints!(constraint_list, x, y, n, sample_num)
  num_thetas = length(x)
  ## 1st exponential cone constraint (zi - ei, s1, t1) in Kexp
  c_start = 1 + num_thetas + (sample_num - 1) * 5 + 1
  A = spzeros(3, n)
  A[1, c_start] = -1.
  y == 1. ? (a = -1) : (a = 1)
  for k = 1:num_thetas
    A[1, 2 + k - 1] = a * x[k]
  end
  A[2, c_start + 3] = 1.
  A[3, c_start + 1] = 1.
  b = zeros(3)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ExponentialCone))

  ## 2nd exponential cone constraint (-e, s2, t2)
  A = spzeros(3, n)
  A[1, c_start] = -1.
  A[2, c_start + 4] = 1.
  A[3, c_start + 2] = 1.
  b = zeros(3)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ExponentialCone))

  ## Nonnegatives constraint t1 + t2 <= 1
  A = spzeros(1, n)
  A[1, c_start + 1] = -1.
  A[1, c_start + 2] = -1.
  b = [1.]
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.Nonnegatives))

  ## ZeroSet constraint s1 == 1, s2 == 1
  A = spzeros(2, n)
  A[1, c_start + 3] = 1.
  A[2, c_start + 4] = 1.
  b = -1 * ones(2)
  push!(constraint_list, COSMO.Constraint(A, b, COSMO.ZeroSet))
end
nothing

# Now we can use this function to loop over the sample points and add the constraints to our constraint list:

#-
constraint_list = Array{COSMO.Constraint{Float64}}(undef, 0)
for i = 1:n
  add_log_regression_constraints!(constraint_list, X[i, :], y[i], nn, i )
end

# It remains to add a second order cone constraint for the regularisation term:
# $\|\theta \|_2 \leq v$

#-
push!(constraint_list, COSMO.Constraint(Matrix(1.0I, n_theta + 1, n_theta + 1), zeros(n_theta + 1), COSMO.SecondOrderCone, nn, 1:n_theta+1));

# We can now create, assemble, and solve our `COSMO.Model`:

#-
model = COSMO.Model()
assemble!(model, P, q, constraint_list, settings = COSMO.Settings(verbose=true))
res = COSMO.optimize!(model);

# Let us double check that we get the same $\theta$ as in the previous section:

#-
using Test
theta_cosmo = res.x[2:2+n_theta-1]
@test norm(theta_cosmo - theta) < 1e-10


"""
    horner(c,x)

Evaluate a polynomial whose coefficients are given in ascending
order in `c`, at the point `x`, using Horner's rule.
"""
function horner(c,x)
    n = length(c)
    y = c[n]
    for k in n-1:-1:1
        y = x*y + c[k]
    end
    return y
end


# The 5-digit number, 16807=7^5, is also a fifth power. Similarly, the 9-digit
# number, 134217728=8^9, is a ninth power.
#
# How many n-digit positive integers exist which are also an nth power?

using ProjectEulerSolutions

# Brute for approach, cycling through nth powers on single digits until we see
# that no more signal digits have the right number of digits (using the built-in
# ndigits function).
function p063solution_bruteforce(max_n::Integer=10)::Integer

    digs = BigInt.(1:9)
    count = 0
    for n = 1:max_n
        c = sum(ndigits.(digs .^ n) .== n)
        if c == 0
           break
        end
        count += c
    end
    return count
end

# Recognize that a single digit number k will have n digits as long as
# 10^(n-1) <= k^n < 10^n.  We know k^n < 10^n when k is 1:9.  Solve for k
# 10^((n-1) / n) <= k and cycle through n values to see the number of values for
# n where the relation holds.  Vectorize and try with k = 1:9 to solve the
# problem in a single line.
function p063solution_explicit_powers(max_n::Integer=10)::Integer
    return Integer(sum(9 .- floor.(10 .^((0:(max_n-1))./(1:max_n)))) + 1)
end

# Alternatively, solve directly for n: 10^(n-1) = k^n, (n-1) log(10) = n log(k),
# n = log(10) / (log(10) - log(k)), where we pass values for k = 1:9.
function p063solution_explicit_logs(max_n::Integer=10)::Integer
    return Integer(sum(floor.(log.(10)./(log.(10).-log.(1:9)))))
end

p063 = Problems.Problem(Dict("Brute-force" => p063solution_bruteforce,
                             "Explicit powers" => p063solution_explicit_powers,
                             "Explicit logs" => p063solution_explicit_logs))

Problems.benchmark(p063, 30)


# L2 norm (times a constant)

export NormL2

"""
**``L_2`` norm**

    NormL2(λ=1.0)

With a nonnegative scalar parameter λ, returns the function
```math
f(x) = λ\\cdot\\sqrt{x_1^2 + … + x_n^2}.
```
"""
struct NormL2{R <: Real} <: ProximableFunction
    lambda::R
    function NormL2{R}(lambda::R) where {R <: Real}
        if lambda < 0
            error("parameter λ must be nonnegative")
        else
            new(lambda)
        end
    end
end

is_convex(f::NormL2) = true

NormL2(lambda::R=1.0) where {R <: Real} = NormL2{R}(lambda)

function (f::NormL2)(x::AbstractArray)
    return f.lambda*norm(x)
end

function prox!(y::AbstractArray{T}, f::NormL2, x::AbstractArray{T}, gamma::Real=1.0) where T <: RealOrComplex
    normx = norm(x)
    scale = max(0, 1-f.lambda*gamma/normx)
    for i in eachindex(x)
        y[i] = scale*x[i]
    end
    return f.lambda*scale*normx
end

function gradient!(y::AbstractArray{T}, f::NormL2, x::AbstractArray{T}) where T <: Union{Real, Complex}
    fx = norm(x) # Value of f, without lambda
    if fx == 0
        y .= 0
    else
        y .= (f.lambda/fx).*x
    end
    return f.lambda*fx
end

fun_name(f::NormL2) = "Euclidean norm"
fun_dom(f::NormL2) = "AbstractArray{Real}, AbstractArray{Complex}"
fun_expr(f::NormL2) = "x ↦ λ||x||_2"
fun_params(f::NormL2) = "λ = $(f.lambda)"

function prox_naive(f::NormL2, x::AbstractArray{T}, gamma::Real=1.0) where T <: RealOrComplex
    normx = norm(x)
    scale = max(0, 1-f.lambda*gamma/normx)
    y = scale*x
    return y, f.lambda*scale*normx
end


using Random
using LinearAlgebra
using PyPlot
using Seaborn
#PyPlot.svg(true)

function iterate_CG(xk, rk, pk, A)
    """
    Basic iteration of the conjugate gradient algorithm
    Parameters:
    xk: current iterate
    rk: current residual
    pk: current direction
    A: matrix of interest
    """

    # form products
    rkk = rk' * rk
    Apk = A * pk

    # construct step size
    αk = rkk / (pk' * Apk)

    # take a step in current conjugate direction
    xk_new = xk + αk * pk

    # construct new residual
    rk_new = rk + αk * Apk

    # construct new linear combination
    betak_new = (rk_new' * rk_new) / rkk

    # generate new conjugate vector
    pk_new = -rk_new + betak_new * pk

    return xk_new, rk_new, pk_new

end

function run_conjugate_gradient(x0, A, b; max_iter = 2000)
    """
    Conjugate gradient algorithm

    Parameters:
    x0: initial point
    A: matrix of interest
    b: vector in linear system (Ax = b)
    max_iter: max number of iterations to run CG
    """

    # initial iteration
    xk = x0
    rk = A * xk - b
    pk = -rk

    err = sum(abs.(rk))
    errors = [err]
    D = pk

    #while sum(abs.(rk)) > 10e-6
    for i in 1:max_iter
        xk, rk, pk = iterate_CG(xk, rk, pk, A)

        D = hcat(D,pk)

        err = sum(abs.(rk))
        push!(errors, err)

        # print iteration
        i % 500 == 0 && println("Iteration $i: error: $err")

        # break if we reach desired error
        if err < 10e-6
            println("Terminated in $i iterations")
            break
        end
    end

    return xk, errors, D
end

Random.seed!(123)
b = rand(5)

A = zeros(5,5)
A[1,1]=10; A[2,2]=10.1; A[3,3]=10.2; A[4,4]=2; A[5,5] = 1

x = A \ b

x0 = [0.3,0.1,0.1,0.1,0.1]
xsol, errors, D = run_conjugate_gradient(x0, A, b);
xsol

Seaborn.semilogy(0:length(errors)-1, errors)
xlabel("iteration"); ylabel("total absolute error")

# confirm these are 0
@assert D[:,1]' * A * D[:,2] < 10e-6
@assert D[:,1]' * A * D[:,3] < 10e-6
@assert D[:,1]' * A * D[:,4] < 10e-6

# confirm these aren't 0
@assert D[:,1]' * A * D[:,1] < 10e-6

# Construct a random symmetric positive definite matrix
Random.seed!(111)
A = rand(5,5); A = 0.5*(A+A'); A = A + 5*I;

eigvals(A)

x2 = A \ b

x0 = [0.,0.,0.,0.,0.]
xsol, errors, D = run_conjugate_gradient(x0, A, b);
xsol

Seaborn.semilogy(0:length(errors)-1, errors)
xlabel("iteration"); ylabel("total absolute error")

N = 1000
Random.seed!(123)
b = rand(N)
A = rand(N,N); A = 0.5*(A+A'); A = A + 50*I;
println("The condition number is: ", cond(A))

# distribution of the eigenvalues
plt[:hist](eigvals(A), bins=500); xlim(0,100)

x0 = zeros(N)
xsol, errors, D = run_conjugate_gradient(x0, A, b);
xsol;

Seaborn.semilogy(0:length(errors)-1, errors)
xlabel("iteration"); ylabel("total absolute error")

D[:,1]' * A * D[:,2]

D[:,1]' * A * D[:,10]

D[:,1]' * A * D[:,33]

Random.seed!(123)

# Generate random s.p.d. matrix
A = rand(N,N); A = 0.5*(A+A'); A = A + 13*I;
println("The condition number is: ", cond(A))

# distribution of the eigenvalues
plt[:hist](eigvals(A), bins=500); xlim(0,100)

x0 = zeros(N)
xsol, errors, D = run_conjugate_gradient(x0, A, b; max_iter=5000);

Seaborn.semilogy(0:length(errors)-1, errors)
xlabel("iteration"); ylabel("total absolute error")

D[:,1]' * A * D[:,2]

D[:,1]' * A * D[:,7]

D[:,1]' * A * D[:,75]


using DataFrames
using DataArrays
using DifferentialEquations
using ParameterizedFunctions
using Gadfly

# definition of differential equations
g = @ode_def_nohes LotkaVolterra begin
         dx = A*x - B*x*y
         dy = -C*y + D*x*y
end A=>1.0 B =>1.0 C =>1.0 D =>1.0

# solving differential equations
function lotka_volterra(a, b, c, d, t0, t1, x0, y0)
  u0 = [x0;y0]
  tspan = (t0,t1)
  h = LotkaVolterra(A=a,B=b,C=c,D=d)
  prob = ODEProblem(h,u0,tspan)
  sol = solve(prob, RK4(), dt=0.01)
  return sol
end

# extracting results and saving into array
function into_array(sol, expNum)
  results = Array{Any,1}[]
  mintime = first(sol.t)
  maxtime = last(sol.t)
  i = mintime
  push!(results, ["t", "x", "y", "experiment"])
  while i < maxtime
    push!(results, [i, sol(i)[1], sol(i)[2], "exp$expNum"])
    i += 0.1
  end
  return results
end

# saving results (array) into csv file
function into_file(filename, results)
  open(filename,"w")
  writecsv(filename, results)
end

function experiments()
  s1 = lotka_volterra(1.5,1.0,3.0,1.0,0.0,5.0,8.0,4.0)
  s2 = lotka_volterra(1.5,1.0,3.0,1.0,0.0,5.0,16.0,2.0)
  s3 = lotka_volterra(1.5,1.0,3.0,1.0,0.0,5.0,12.0,4.0)
  s4 = lotka_volterra(1.5,1.0,3.0,1.0,0.0,5.0,2.0,10.0)
  into_file("res1.csv",into_array(s1,1))
  into_file("res2.csv",into_array(s2,2))
  into_file("res3.csv",into_array(s3,3))
  into_file("res4.csv",into_array(s4,4))
  return [s1,s2,s3,s4]
end

# read in csv files into DataFrame
function read_in(res...)
  df = vcat([readtable(res[i]) for i in 1:length(res)])
end

# print out min, max and mean value for each experiment
function min_max_mean(data)
  experiments = sort(unique(data[:experiment]))
  i = 1
  for exp in experiments
    println("Experiment $i\n")
    exp_data = data[data[:experiment] .== exp, :]
    println("Prey population\n")
    println("Minimum: $(minimum(exp_data[:x]))")
    println("Maximum: $(maximum(exp_data[:x]))")
    println("Mean:    $(mean(exp_data[:x]))")
    println("\nPredator population\n")
    println("Minimum: $(minimum(exp_data[:y]))")
    println("Maximum: $(maximum(exp_data[:y]))")
    println("Mean:    $(mean(exp_data[:y]))")
    println("\n")
    i += 1
  end
end

# adding new column
function diff_column(data)
  new_data = data
  new_data[:diff] = new_data[:y] - new_data[:x]
  return new_data
end

# plotting functions
function plot_prey_population(sol)
  df=DataFrame(t=sol.t, prey=map(x->x[1],sol.u))
  plot(df,  x="t", y="prey") #prey population
end

function plot_predator_population(sol)
  df2=DataFrame(t=sol.t, predator=map(y->y[2],sol.u))
  plot(df2,  x="t", y="predator") #predator population
end

function plot_prey_predator(data)
  dataframe = DataFrame()
  exp1_data = data[data[:experiment] .== "exp1", :]
  exp2_data = data[data[:experiment] .== "exp2", :]
  exp3_data = data[data[:experiment] .== "exp3", :]
  exp4_data = data[data[:experiment] .== "exp4", :]
  dataframe[:x1] = exp1_data[:x]
  dataframe[:y1] = exp1_data[:y]
  dataframe[:x2] = exp2_data[:x]
  dataframe[:y2] = exp2_data[:y]
  dataframe[:x3] = exp3_data[:x]
  dataframe[:y3] = exp3_data[:y]
  dataframe[:x4] = exp4_data[:x]
  dataframe[:y4] = exp4_data[:y]

  plot(dataframe, layer(x="x1", y="y1", Geom.point, Theme(default_color= colorant"blue")),
                  layer(x="x2", y="y2", Geom.point, Theme(default_color= colorant"red")),
                  layer(x="x3", y="y3", Geom.point, Theme(default_color= colorant"green")),
                  layer(x="x4", y="y4", Geom.point, Theme(default_color= colorant"yellow")),
                  Guide.XLabel("Prey population"),
                  Guide.YLabel("Predator population"),
                  Guide.Title("Lotka - Volterra"))
end

solutions = experiments()

solutions_data = read_in("res1.csv", "res2.csv", "res3.csv", "res4.csv")

min_max_mean(solutions_data)

diff_column(solutions_data)

plot_prey_population(solutions[1])

plot_predator_population(solutions[1])

plot_prey_predator(solutions_data)


export cgs, cgs!, mgs, mgs!

"""
KrylovMethods.cgs = KrylovMethods.cgs!(copy(V))

	Classical Gram Schmidt orthogonalization.

"""
cgs(V) = cgs!(copy(V))

"""
function KrylovMethods.cgs!

Inplace Classical Gram Schmidt orthogonalization.

Reference: page 254 in Golub and Van Loan, Matrix Computation, 4th edition.

Input:

	V::Array  - m by n matrix of full rank m<=n

Output:

	V::Array  - m-by-m unitary matrix
	R::Array  - m-by-n upper triangular matrix

"""
function cgs!{T}(V::Array{T})

	m,n    = size(V)
	R      = zeros(T,n,n)
	R[1,1] = norm(V[:,1])
	V[:,1] ./= R[1,1]
	for k=2:n
	    R[1:k-1,k] = V[:,1:k-1]'*V[:,k]
	    V[:,k]     = V[:,k]-V[:,1:k-1]*R[1:k-1,k]
	    R[k,k]     = norm(V[:,k])
	    V[:,k]   ./= R[k,k]
	end
	return V,R
end

"""
KrylovMethods.mgs = KrylovMethods.mgs!(copy(V))

	Modified Gram Schmidt orthogonalization.

"""
mgs(V) = mgs!(copy(V))

"""
function KrylovMethods.mgs!

Inplace Modified Gram Schmidt orthogonalization.

Reference: page 255 in Golub and Van Loan, Matrix Computation, 4th edition.

Input:

	V::Array  - m by n matrix of full rank m<=n

Output:

	V::Array  - m-by-m unitary matrix
	R::Array  - m-by-n upper triangular matrix

"""
function mgs!{T}(V::Array{T})
	m,n    = size(V)
	R      = zeros(T,n,n)
	for k=1:n
		R[k,k]    = norm(V[:,k])
		V[:,k]    ./= R[k,k]
		for j=k+1:n
			R[k,j] = dot(V[:,k],V[:,j])
			V[:,j] -= R[k,j]*V[:,k]
		end
	end
	return V,R
end


using Distributions, Plots; pyplot()

alpha, beta = 8, 2
prior(lam) = pdf(Gamma(alpha, 1/beta), lam)
data = [2,1,0,0,1,0,2,2,5,2,4,0,3,2,5,0]

like(lam) = *([pdf(Poisson(lam),x) for x in data]...)
posteriorUpToK(lam) = like(lam)*prior(lam)

delta = 10^-4.
lamRange = 0:delta:10
K = sum([posteriorUpToK(lam)*delta for lam in lamRange])
posterior(lam) = posteriorUpToK(lam)/K

bayesEstimate = sum([lam*posterior(lam)*delta for lam in lamRange])

newAlpha, newBeta = alpha + sum(data), beta + length(data)
closedFormBayesEstimate = mean(Gamma(newAlpha, 1/newBeta))

println("Computational Bayes Estimate: ", bayesEstimate)
println("Closed form Bayes Estimate: ", closedFormBayesEstimate)

plot(lamRange, prior.(lamRange),
	c=:blue, label="Prior distribution")
plot!(lamRange, posterior.(lamRange),
	c=:red, label="Posterior distribution",
	xlims=(0, 10), ylims=(0, 1.2),
    	xlabel=L"\lambda",ylabel="Density")


"Square the sum of the first `n` positive integers"
function square_of_sum(n)
    (n * (n + 1) ÷ 2)^2
end

"Sum the squares of the first `n` positive integers"
function sum_of_squares(n)
    n * (n + 1) * (2 * n + 1) ÷ 6
end

"Subtract the sum of squares from square of the sum of the first `n` positive ints"
function difference(n)
    square_of_sum(n) - sum_of_squares(n)
end


acknowledge help from Tommy Hofmann

****************************
linearcode.jl
****************************
include("linearcode.jl");

# To create a linear code, need to make generator matrix. We start by creating a field for this.
F, _ = FiniteField(2, 1, "α");
# one may either create a matrix space and then embedd the matrix into it as in
M = MatrixSpace(F, 4, 7);
G = M([1 0 0 0 0 1 1;
       0 1 0 0 1 0 1;
       0 0 1 0 1 1 0;
       0 0 0 1 1 1 1]);
# or use either of the two shorthand notations
G2 = matrix(F, [1 0 0 0 0 1 1;
       0 1 0 0 1 0 1;
       0 0 1 0 1 1 0;
       0 0 0 1 1 1 1]);
G3 = matrix(F, 4, 7, [1, 0, 0, 0, 0, 1, 1, 0, 1, 0, 0, 1, 0, 1, 0, 0, 1, 0, 1, 1, 0, 0, 0, 0, 1, 1, 1, 1]);
G == G2
G == G3

# This is the generator matrix for the [7, 4, 3] binary Hamming code in standard form
C = LinearCode(G)
# We can get the basic information
length(C)
dimension(C)
cardinality(C)
rate(C)
# since we gave it a full rank matrix, the rank should equal the dimension
rank(G) == dimension(C)
# no minimum distance calculations were performed so the object so far does not know
# this value and the current state of the variable is set to 'missing'
# functions which rely on this value will report an error in this case
minimumdistance(C)
relativedistance(C)
genus(C)
# since we know the minimum distance of this code is 3, we can go ahead and set that
# no checking is done to verify the user's input
# since setting the minimum distance is a desirable thing to be able to do, all code
# objects are mutable (and hence live on the heap)
setminimumdistance!(C, 3)
minimumdistance(C)
relativedistance(C)
genus(C)
isMDS(C)
numbercorrectableerrors(C)
# can check the generator and parity check matrices and check their defining properties
GC = generatormatrix(C)
G == GC
H = paritycheckmatrix(C)
iszero(G * H')
iszero(H * G')
# if G is full rank, the matrix is stored directly as the generator matrix of C as written
# and H is computed as the right kernel of G
# the standard form of G and H may be retrieved by passing true into the optional parameter
Gstand = generatormatrix(C, true)
Hstand = paritycheckmatrix(C, true)
# G in the example was already in standard form, so we can check the computations
Gstand == GC
Hstand == H
# finally, we may compute the dual
# in order to save computational time, this command merely flipps the roles of G and H
# in the LinearCode constructor, and no new computation is done
dual(C)

# a vector v is in the code C if it has zero syndrome
r = generatormatrix(C)[1, :]
iszero(syndrome(r, C))
# a code C1 is a subset of C2 if every row of the generator matrix of C1 is in C2
C ⊆ C
C ⊆ dual(C)
issubcode(C, dual(C))
# two codes C1 and C2 are equivalent if C1 ⊆ C2 and C2 ⊆ C1
isequivalent(C, dual(C))
# a code is self dual if it is equivalent to its dual
isselfdual(C)
# code is self orthogonal if it is a subcode of its dual
isselforthogonal(C)

# finally, one may encode a length k vector by C using
encode(C.G[:, 1], C)
# here the vector to be encoded is an element of MatrixSpace(Nemo.GF(2), 4, 1)
# allowing for proper matrix multiplication between it and the generator matrix
# one may also pass in a Vector{Int64} of the proper length and the appropriate
# matrix space will automatically be created. this may be slow if this is done
# repeatedly as this object will be built from scratch each time. the same is
# true for syndrome(). either a (1, C.k) or a (C.k, 1) dimensional vector may
# be used as the code will automatically apply the transpose if necessary
# it will not however accept objects of type adjoint such as v'
generatormatrix(C)[:, 1]
v = [1, 0, 0, 0];
encode(v, C)
v2 = [1; 0; 0; 0];
encode(v2, C)
generatormatrix(C)[1, :]
v = [1, 0, 0, 0, 0, 1, 1];
syndrome(v, C)
v = [1; 0; 0; 0; 0; 1; 1];
syndrome(v, C)
# might want to remove these transpose examples and remove from code since these have same size

# need examples, not full rank
# passing in H
# passing in G and H directly

# we can preform several standard methods of building new codes from old ones
# direct sum
D = C ⊕ C;
generatormatrix(D)
D = directsum(C, C);
generatormatrix(D)
# direct product
D = C ⊗ C;
generatormatrix(D)
D = directproduct(C, C);
generatormatrix(D)
D = tensorproduct(C, C);
generatormatrix(D)
# the most common form of extending a code is to add parity check column to right of generator matrix
D = extend(C);
length(D)
generatormatrix(D)
F3, _ = FiniteField(3, 1, "α");
# M2 = MatrixSpace(Nemo.GF(3), 2, 4); #- ex page 15
G2 = matrix(F3, [1 0 1 1; 0 1 1 -1]);
C2 = LinearCode(G2);
generatormatrix(extend(C2))
paritycheckmatrix(extend(C2))
originalgeneratormatrix(C2) == G2
# in cases such as this it is often useful to keep track of the original generator matrix which
# was used to generate the code. can also do this for the parity check matrix. in cases where this
# does not make sense, the normal generator or parity check matrix is returned
originalgeneratormatrix(D)
originalgeneratormatrix(D) == generatormatrix(D)[:, 1:end-1]
paritycheckmatrix(D)
originalparitycheckmatrix(D)
# puncturing deletes columns from a generator matrix
C2 = puncture(D, [length(D)]);
isequivalent(C, C2)
generatormatrix(C) == generatormatrix(C2)
originalgeneratormatrix(C2) == generatormatrix(C)
# to expurgate a code is to delete rows from generator matrix and then remove any potentially resulting zero columns
D = expurgate(C, [dimension(C)]);
length(D)
dimension(D)
generatormatrix(D)
originalgeneratormatrix(D) == generatormatrix(C)
# to augment is to add rows to the generator matrix
# in this example we are going to first add the all 1's vector to the code, which is common
# we note that the all 1's vector is actually already in the code so the code
# shouldn't change
# M = MatrixSpace(Nemo.GF(2), 1, 7);
v = matrix(F, [1 for i in 1:7]')
v ∈ C
D = augment(C, v);
generatormatrix(D) == generatormatrix(C)
v = matrix(F, [1 0 1 0 1 1 1])
v ∈ C
D = augment(C, v);
generatormatrix(D)
originalgeneratormatrix(D) == generatormatrix(C)
# to shorten is to expurgate then puncture
D = shorten(C, [length(C)]);
generatormatrix(D)
originalgeneratormatrix(D) == generatormatrix(C)
# M2 = MatrixSpace(Nemo.GF(2), 3, 6);
G2 = matrix(F, [1 0 0 1 1 1; 0 1 0 1 1 1; 0 0 1 1 1 1]);
C2 = LinearCode(G2);
generatormatrix(shorten(C2, [5, 6]))
# the most common form of lengthening a code it to augment the all 1's vector and then extend
D = lengthen(C);
generatormatrix(D)
originalgeneratormatrix(D) == generatormatrix(C)
# the (u | u + v)- or Plotkin construction
# M2 = MatrixSpace(Nemo.GF(2), 3, 4);
# M3 = MatrixSpace(Nemo.GF(2), 1, 4);
G2 = matrix(F, [1 0 1 0; 0 1 0 1; 0 0 1 1]);
G3 = matrix(F, [1 1 1 1]);
C2 = LinearCode(G2);
C3 = LinearCode(G3);
generatormatrix(uuplusv(C2, C3)) #- ex p. 19

SingletonBound

****************************
cyclotomic.jl
****************************
include("cyclotomic.jl")

# ord_n(q)
q = 2;
n = 15;
b = 3;
δ = 4;
ord(n, q)
# at the moment this comes back sorted so one can easily record a coset representative
coset = cyclotomiccoset(3, q, n, false)
cyclotomiccoset(3, q, n, true)
allcyclotomiccosets(q, n, true)
dualqcosets(q, n, [coset])
complementqcosets(q, n, [coset])
# pairings
qcosetpairings(q, n)
# find all in a range
qcosettable(10, 13, q)

****************************
cycliccode.jl
****************************
include("cycliccode.jl")
q = 2;
n = 15;
b = 3;
δ = 4;
cosets = definingset([i for i = b:(b + δ - 2)], q, n, false)
C = CyclicCode(q, n, cosets)
basefield(C)
splittingfield(C)
polynomialring(C)
primitiveroot(C)
offset(C)
designdistance(C)
qcosets(C)
qcosetsreps(C)
definingset(C)
generatorpolynomial(C)
paritycheckpolynomial(C)
idempotent(C)
# CyclicCode is a subtype of LinearCode so the functions of the previous section also apply here
originalgeneratormatrix(C)
G = matrix(F, [1 0 0 0 0 1 1;
       0 1 0 0 1 0 1;
       0 0 1 0 1 1 0;
       0 0 0 1 1 1 1]);
D = LinearCode(G);
CD = directsum(C, D);
generatormatrix(CD)
# the generator matrix of a cyclic code is kept in cyclic form but the standard form is of course always available
generatormatrix(C)
generatormatrix(C, true)
# note how the previous code auto detected that it was a BCH code and returned an object of type BCHCode
complement(C)
# the above is equivalent to
B = BCHCode(q, n, δ, b)
C == B
# the optional offset parameter is set to 0 by default
BCHCode(q, n, δ)
# for cyclic codes, we can
C ∩ B == C
B = BCHCode(q, n, δ - 1, b + 4)
D = C ∩ B # check later that this is == repetition code
C + B

# Reed Solomon codes are BCH codes with n = q^m - 1
# this forces the cyclotomic cosets to each have size one
q = 16;
n = 15;
b = 3;
δ = 4;
allcyclotomiccosets(q, n, true)
# as with BCH codes, it will auto detect whether or not the code is Reed Solomon and call the appropriate constructor
cosets = definingset([i for i = b:(b + δ - 2)], q, n, false);
CyclicCode(q, n, cosets)
BCHCode(q, n, δ, b)

############## this doesn't work
ReedSolomonCode(q, δ, b)

come up with better examples from the book but basis functionality is working so far and interaction with LinearCode is good
fix C ∩ complement(C) example where defining set is an extreme (Vector{Any} so probably empty)
2nd form of CyclicCode constructor

****************************
ReedMuller.jl
****************************
include("ReedMuller.jl")

# so far only binary implemented
# fields are created via F, _ = FiniteField(2, 1, "α") by default
# codes are created using the recursive form of the generator matrix
q = 2;
ReedMullergeneratormatrix(q, 0, 3)
ReedMullergeneratormatrix(q, 2, 2)
ReedMullerCode(q, 1, 2)
C = ReedMullerCode(q, 1, 3)
ReedMullerCode(q, 2, 3)
isselfdual(C)


# Example driver for the geometric multigrid program, GMG.jl, to solve a Poisson equation on a 2D grid using
#  second order finite difference method.
#  Dirichlet Boundary conditions

include("GMG.jl")
using .GMG: V_cycle!
using Formatting: printfmt

# Define the problem being solved. It is on a 1x1 square domain with
# homogeneous dirichlet boundary conditions
function U_exact(x,y)
  return (x^3-x)*(y^3-y)
end

function source(x,y)
  return 6*x*y*(x^2 + y^2 -2)
end

max_cycles = 10              # max number of V cycles to execute
nlevels    = 10              # number of grid levels. 1 means no multigrid, 2 means one coarse grid. etc
nx         = 1*2^(nlevels-1) # Nx and Ny are given as function of grid levels
ny         = 1*2^(nlevels-1) #

const u_ex = zeros(Float64,(nx+2,ny+2))
const u = zeros(Float64,(nx+2,ny+2))
const f = zeros(Float64,(nx+2,ny+2))
const res = zeros(Float64,(nx+2,ny+2))
const error = zeros(Float64,(nx,ny))

#calcualte the RHS and exact solution
DX=1.0/nx
DY=1.0/ny
xc=range(0.5*DX,1-0.5*DX,length=nx)
yc=range(0.5*DY,1-0.5*DY,length=ny)

#set the exact solution and the source term corresponding to it
for i= 1:nx, j=1:ny;
  u_ex[i+1,j+1] = U_exact(xc[i],yc[j])
  f[i+1,j+1]    = source(xc[i],yc[j])
end

println("-------------------------New Run------------------------------")
println("           Poisson equation on $(nx)X$(ny) grid")

for it=1:max_cycles

  u[:,:],res[:,:]=V_cycle!(nx,ny,nlevels,u,f,1)

  error[:,:]=u_ex[2:nx+1,2:ny+1]-u[2:nx+1,2:ny+1]

  printfmt( "|V cycle: {:2d}| res. norm: {:16.9e}| error norm: {:16.9e}|\n", it,findmax(abs.(res))[1],findmax(abs.(error))[1] )
end